Compare commits

..

78 Commits

Author SHA1 Message Date
renovate[bot] cc87e88f03 fix(deps): update module gorm.io/driver/mysql to v1.6.0 (#819)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-08-07 23:20:41 +08:00
Ovear 7e9082f90e fix(stream): allow seeking beyond file size per Go convention (#2667)
Remove `offset > size` guard from RangeReadReadAtSeeker.Seek and
DynamicReadAtSeeker.Seek. Go's io.Seeker convention allows seeking
past EOF.

Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-08-07 22:53:49 +08:00
renovate[bot] 6d4ad58678 fix(deps): update module golang.org/x/net to v0.57.0 (#2748)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-08-07 22:50:11 +08:00
renovate[bot] d8ad28f5b7 fix(deps): update module github.com/ipfs/go-cid to v0.6.2 (#2738)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-08-07 22:44:19 +08:00
renovate[bot] 2e27852952 fix(deps): update azure-sdk-for-go monorepo (#2700)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-08-07 22:43:31 +08:00
renovate[bot] 194f941f15 fix(deps): update module golang.org/x/sys to v0.47.0 (#2751)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-08-07 22:42:26 +08:00
renovate[bot] ec1f968de1 fix(deps): update module github.com/shirou/gopsutil/v4 to v4.26.7 (#2896)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-08-07 22:41:10 +08:00
renovate[bot] c97173286c fix(deps): update module github.com/nwaples/rardecode/v2 to v2.3.0 (#2898)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-08-07 22:40:45 +08:00
ThetaPilla. cca773f07b feat(fs): add pipelined multipart upload (#2723)
* feat(multipart): add chunk reassembly window

- Reassemble concurrently uploaded chunks into a sequential stream through a ring file, bounding disk usage to slots*chunkSize per session
- Park writers up to a deadline when their slot is busy instead of rejecting instantly, so flow control does not surface as connection errors in browsers
- Record a per-chunk CRC32 table for re-fill verification and keep it readable after close
- Propagate cancellation to blocked readers via CloseWithError so drivers treat aborts like canceled requests
- Cover ordering, backpressure, idempotent resends and close/abort wake-ups with race-enabled tests

* feat(multipart): add pipelined upload session manager

- Start the driver upload at session init over a sequential stream backed by the window, so client-to-server and server-to-storage transfers run concurrently
- Attach client-provided hashes to the stream so drivers can attempt rapid upload before any chunk arrives, and absorb chunks racing pipeline completion idempotently
- Keep only metadata and chunk CRCs after a failed attempt: re-sending chunk 0 re-fills a fresh window, and content changes between attempts are rejected
- Resume receiving sessions only when client hashes prove the same file; failed_retriable sessions resume unconditionally
- Reclaim sessions with a sliding-TTL GC and sweep orphaned ring files at startup
- Cover the state machine with race-enabled tests over a stubbed storage layer

* feat(setting): add multipart upload settings

- Add multipart_enabled and multipart_chunk_size (MB) as public traffic settings
- Validate the chunk size on save (integer within 1-90) via the setting item hook

* feat(server): add multipart upload API

- Add /api/fs/multipart init/chunk/complete/status/abort endpoints with headers aligned with /fs/put
- Gate init behind the FsUp permission checks and reuse the client upload rate limiter for chunk uploads
- Drain the request body before answering chunk requests on every path, so browsers do not see early responses as network errors
- Start the multipart session GC when the router is initialized to reclaim ring files orphaned by a previous run

* refactor(multipart): remove unused overwrite session field

- The overwrite flag was stored on the session but never read: the handler
  performs the pre-check and op.Put owns the overwrite semantics

* feat(setting): allow any positive multipart chunk size

- Validate the setting as a positive integer only; self-hosted admins decide
  the ceiling themselves instead of an arbitrary 90MB cap
- Keep clamping the client-suggested X-Chunk-Size to the admin value: the
  server buffers a window of 8 chunks per session, so an unbounded client
  suggestion would translate directly into server-side disk usage

* refactor(server): simplify multipart chunk size clamp

- Fold the two-step clamp into one branch: the ceiling is already floored,
  so a client suggestion just lowers the size with a 1MB floor
2026-08-06 14:14:35 +08:00
renovate[bot] a4ae2acb65 fix(deps): update module github.com/sheltonzhu/115driver to v1.3.5 (#2895)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-08-04 12:03:42 +08:00
MadDogOwner f68f3e9f4a fix(drivers/guangyapan): remove default value (#2894)
* fix(drivers/guangyapan): remove default value for ClientID

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/guangyapan): remove unused configuration options from driver settings

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-08-04 11:53:45 +08:00
Pikachu Ren 6f80df2827 feat(drivers): add GuangYaPan driver with offline download support (#2882)
* feat(drivers): add GuangYaPan driver with offline download support

* fix(drivers): GuangYaPan driver multipartUploadToOSS
2026-08-04 11:34:56 +08:00
没空的人 940b69fe38 fix(drivers/lanzou): handle acw_sc__v2 challenge in Login function (#2876)
The Login endpoint may return an acw_sc__v2 validation page when accessed,
which previously caused login failures. This change adds the same retry
logic and cookie handling already used in request() to automatically solve
the challenge, ensuring login success even when the anti-bot mechanism is
triggered.
2026-07-31 18:13:47 +08:00
Yinan Qin f2f7ee4107 fix(ci): change missed pull_request to pull_request_target (#2873) 2026-07-30 00:45:01 +08:00
ILoveScratch cf931d4760 chore: fix typos (#2859) 2026-07-28 13:18:34 +08:00
ILoveScratch be708292eb chore: fix up .gitignore (#2857) 2026-07-28 07:37:07 +08:00
Liangbin Lian 78f2f0b25d chore(deps): remove unimplemented FUSE mount functionality and dependencies (#2854)
FUSE was introduced years ago but never implemented,
let's remove it to reduce dependencies.

Signed-off-by: Liangbin Lian <jjm2473@gmail.com>
2026-07-28 01:50:22 +08:00
Yinan Qin 6b92cbd21f fix(ci): allow pr check to run by default (#2856) 2026-07-27 23:52:20 +08:00
ShenLin 84ecda35aa fix(search): apply access filtering before paginating results
* fix: replace strings.HasPrefix with utils.IsSubPath for path validation

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix: re-validate shared paths to ensure they remain within the creator's base path

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(search): apply access filtering before paginating results

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
2026-07-23 20:15:21 +08:00
renovate[bot] 2deb16b26e fix(deps): update github.com/t3rm1n4l/go-mega digest to c6acd6a (#2829)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-23 19:44:46 +08:00
ShenLin 8495470e63 fix(search): serialize index updates by parent (#2827)
- Prevent concurrent updates from inserting duplicate index entries
- Preserve parallel indexing across different parent paths
- Add serialization, concurrency, and lock cleanup tests

Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-07-23 19:33:25 +08:00
ShenLin fe56ca483d fix(drivers/s3): support multipart object copy (#2826)
fix(s3): support multipart object copy

- Route objects larger than 5 GB through multipart copy
- Preserve source metadata and abort incomplete multipart copies
- Cover size routing, byte ranges, metadata, and failure cleanup

Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-07-23 19:33:16 +08:00
ShenLin e83835dd80 fix(buffer): wait for active I/O before closing (#2825)
* fix(buffer): wait for active I/O before closing

- Track in-flight PipeBuffer read and write operations
- Wait for active I/O before releasing backing storage
- Add concurrent close regression coverage for reads and writes

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>

* ensure waitgroup handling

---------

Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-07-23 19:33:14 +08:00
Firmant Xu 791a27e2ac feat(drivers): add Bunny Storage driver (#2472)
* feat(driver/bunny_storage): add Bunny Storage driver

Add Bunny Storage API support with optional CDN links and token signing.

Cover CDN mount path handling, token signing, and proxy/cache behavior with unit tests.

* replace RootFolderPath with driver.RootPath in Addition struct

* fix(driver/bunny_storage): parse fractional Bunny timestamps

* fix(bunny_storage): correct CDN token signing

- Decode URL paths without treating literal plus signs as spaces.
- Reject duplicate query parameters and cover both cases with regression tests.

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>

---------

Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
2026-07-23 19:29:53 +08:00
ShenLin 28cea1fb2b fix(proxy): preserve proxied download filenames (#2824)
- Override transparent proxy Content-Disposition with the OpenList object name
- Add regression coverage for incorrect upstream download headers

Signed-off-by: jyxjjj <16695261+jyxjjj@users.noreply.github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
2026-07-23 16:10:45 +08:00
So 0a9e71ae67 feat(drivers/139): merge multiple share roots (#2783)
* feat(drivers/139): merge multiple share roots

* fix(drivers/139): fallback to share preview without account

* chore(drivers/139): address share review comments

* refactor(drivers/139): reuse crypto

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* docs(drivers/139): add share support

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: syscc <syscc@users.noreply.github.com>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2026-07-23 10:49:54 +08:00
So a21bc8bd0f fix(115_open): clean parent path before lookup (#2820) 2026-07-22 22:21:00 +08:00
MadDogOwner bacd087096 refactor(drivers/139): update path handling, put for family/group (#2719)
* fix(drivers/139): update path handling for family

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* perf(drivers/139): add retry go for family upload

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* feat(drivers/139): add FamilyCloudHost and GroupCloudHost handling

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* perf(drivers/139): add retry go for personalnew upload

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/139): use new batchpartinfos for remaining parts

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(driver/139): do not use path in id

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(driver/139): refactor RootPath handling

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(driver/139): remove root path stripping in Init method

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* chore(drivers/139): reduce upload retry attempts to 3

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/139): add rate limiting for MetaPersonalNew upload

- Wrap stream with LimitedUploadStream before creating StreamSectionReader
- Aligns with the same pattern used in the MetaPersonal/MetaGroup/MetaFamily path

Co-authored-by: GitHub Copilot <copilot@github.com>

* fix(drivers/139): fix section reader leak and seek check in retry block

- Check rd.Seek() return value and free the section reader on error
- Call ss.FreeSectionReader(rd) on all error paths within retry.Do closure
- Prevents buffer.Block leak when retrying failed upload chunks

Co-authored-by: GitHub Copilot <copilot@github.com>
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/139): move GetSectionReader outside retry loop

- Move ss.GetSectionReader() before retry.Do() so the sequential
  stream section reader is only called once per chunk
- Remove redundant ss.FreeSectionReader() calls from inside the
  retry closure since the reader is now owned by the outer scope
- Fixes 'stream not cached' errors when retrying failed chunk
  uploads during cross-storage WebDAV COPY operations

Co-authored-by: GitHub Copilot <copilot@github.com>

* feat(drivers/139): add UseOldStreamUpload option

- Add UseOldStreamUpload option
- Implement newRequest, newPost
- Support rapid upload for PersonalNew and Group/Family

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/139): add more param for group/family put

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/139): correct group params for new put

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/139): update personalPost to use getPersonalCloudHost

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* chore(drivers/139): correct typo

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/139): pass full partInfos slice for batched upload

- Pass the global partInfos slice instead of the batch subset to
  uploadPersonalParts, so that PartNumber-1 indexing does not go
  out of bounds when a file has more than 100 parts.

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>

* fix(drivers/139): avoid double-counting progress on upload retries

- Save p.Done before each part and reset it inside the retry function
  so that bytes from failed attempts are not counted toward progress.
- Each part is counted exactly once, on the successful attempt.

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>

* fix(drivers/139): gate route-host checks by cloud type

- Only validate GroupCloudHost and FamilyCloudHost for MetaGroup
  and MetaFamily storage types, so that personal-only accounts do
  not fail initialization when the route policy omits group/family.

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>

* fix(drivers/139): add ProviderRoot back for groupgetfiles

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/139): set path to 0 when group old upload

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/139): improve error handling for family root path retrieval

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* refactor(drivers/139): update condition checks for CloudHost initialization

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: GitHub Copilot <copilot@github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
2026-07-22 15:14:38 +08:00
Suyunjing e1ded83d5f fix(drivers/115_open): recover transient path metadata (#2816)
Recover temporarily missing or incomplete 115 Open path metadata from parent
and ancestor directory listings. Use 115-sdk-go v0.2.6 to normalize missing
object errors, while preserving valid zero-byte files and ensuring recursive
parent resolution progresses correctly.

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-22 12:59:41 +08:00
renovate[bot] ff3cb8b931 chore(deps): update actions/setup-go action to v7 (#2798)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 19:50:52 +08:00
Seven 7aea393a6d fix(drivers/115_open): populate file size when resolving paths (#2797)
* fix(115_open): populate file size when resolving paths

* feat(drivers/115_open): update 115-sdk-go
2026-07-16 19:46:15 +08:00
renovate[bot] 36168c86a0 fix(deps): update module github.com/rclone/rclone to v1.74.4 (#2744)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 15:53:43 +08:00
Shelton Zhu 24332e371f fix(115,115share): update github.com/SheltonZhu/115driver to v1.3.4 (#2794)
fix(115,115share): support empty User-Agent for UA-free CDN download links

Fixes #SheltonZhu/115driver/issues/80
2026-07-16 09:49:29 +08:00
renovate[bot] 6b702afc0f fix(deps): update module gorm.io/gorm to v1.31.2 (#2758)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 00:50:53 +08:00
renovate[bot] 78db0f7aff chore(deps): update docker/setup-qemu-action action to v4 (#2765)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 00:35:50 +08:00
renovate[bot] e469693114 fix(deps): update module github.com/bodgit/sevenzip to v1.6.5 (#2789)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 00:35:28 +08:00
renovate[bot] 5961289b83 fix(deps): update module github.com/nwaples/rardecode/v2 to v2.2.5 (#2699)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 00:29:25 +08:00
renovate[bot] 0527dc41af fix(deps): update module github.com/deckarep/golang-set/v2 to v2.9.0 (#2712)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 00:24:18 +08:00
renovate[bot] b1d298d192 fix(deps): update module github.com/go-resty/resty/v2 to v2.17.2 (#2718)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 00:20:11 +08:00
renovate[bot] 651b5cea1c fix(deps): update module github.com/protonmail/gopenpgp/v2 to v2.10.0 (#2742)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 00:11:32 +08:00
renovate[bot] 5aeaf7a8c8 fix(deps): update module github.com/shirou/gopsutil/v4 to v4.26.6 (#2745)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-16 00:08:16 +08:00
ShenLin 651da18da4 fix(handles): add src_name validation in FsBatchRename
- Validate source names with checkRelativePath before batch renames
- Reject invalid source paths with a forbidden response

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2026-07-15 16:11:57 +08:00
MadDogOwner 5a5d8d6e0e refactor(drivers/github_releases): remove internal caching and add pagination support (#2775)
* refactor(github_releases): remove internal caching, use global cache instead

- Remove Release/Releases/OtherFile cache fields from MountPoint struct
- Convert instance methods to pure functions (releaseToFiles, releasesToVersionDirs, etc.)
- Make List() fetch fresh data from GitHub API on every call
- Add githubGet helper to centralize GitHub API requests
- Remove unused GetRequest method and IsAncestorDir function

Co-authored-by: GitHub Copilot <copilot@github.com>

* feat(github_releases): add pagination and max page limit for all versions

- Add PerPage config to control releases per page (default 30, max 100)
- Add MaxPage config to limit max pages fetched (0 = unlimited)
- Rewrite getAllReleases to support automatic pagination

Co-authored-by: GitHub Copilot <copilot@github.com>

* chore(github_releases): move utils from driver.go to util.go

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* chore(github_releases): revert changes to GetRequest

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* chore(github_releases): use more common format for github proxy url

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(github_releases): show_readme in empty releases case

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(github_releases): preserve README and LICENSE name variants

Co-authored-by: Copilot <copilot@github.com>
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: GitHub Copilot <copilot@github.com>
2026-07-15 14:19:56 +08:00
renovate[bot] 061c1e2e1e chore(deps): update docker/setup-buildx-action action to v4 (#2764)
[no ci]
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-14 14:30:49 +08:00
renovate[bot] caed356200 chore(deps): update docker/metadata-action action to v6 (#2763)
[no ci]
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-14 14:30:24 +08:00
renovate[bot] 74af297471 chore(deps): update docker/login-action action to v4 (#2762)
[no ci]
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-14 14:29:43 +08:00
renovate[bot] 7c54b0e3e4 chore(deps): update docker/build-push-action action to v7 (#2761)
[no ci]
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-14 14:29:10 +08:00
renovate[bot] a632356902 chore(deps): update benjlevesque/short-sha action to v4 (#2760)
[no ci]
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-14 14:25:01 +08:00
renovate[bot] 82ea8d0d01 chore(deps): update actions/cache action to v6 (#2759)
[no ci]
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-14 14:18:52 +08:00
MadDogOwner eb48671202 fix(drivers/s3): custom user agent and method fixes (#2780)
* feat(drivers/s3): support custom ua

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drivers/s3): add context to copy and remove file methods

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix(drives/s3): avoid double-prepending root path

Co-authored-by: Copilot <copilot@github.com>
Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: Copilot <copilot@github.com>
2026-07-11 11:46:09 +08:00
renovate[bot] f3e74fbdf1 fix(deps): update module github.com/quic-go/quic-go to v0.60.0 (#2743)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-10 12:24:49 +08:00
renovate[bot] e670322ee3 fix(deps): update module github.com/maruel/natural to v1.3.0 (#2739)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-10 12:24:49 +08:00
renovate[bot] 8930e1945e fix(deps): update module google.golang.org/grpc to v1.82.0 (#2754)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-10 12:24:49 +08:00
renovate[bot] daaba0d7ea fix(deps): update module gorm.io/driver/postgres to v1.6.0 (#2756)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-10 12:24:48 +08:00
renovate[bot] a81f54488f fix(deps): update module gorm.io/driver/sqlite to v1.6.0 (#2757)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-10 12:24:48 +08:00
ShenLin 59bd343140 fix(sharing): enforce base path boundaries
* fix: replace strings.HasPrefix with utils.IsSubPath for path validation

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

* fix: re-validate shared paths to ensure they remain within the creator's base path

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>

---------

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
Co-authored-by: MadDogOwner <xiaoran@xrgzs.top>
2026-07-10 12:24:40 +08:00
renovate[bot] 757f9d383d fix(deps): update module github.com/dlclark/regexp2 to v1.12.0 (#2713)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-08 18:35:18 +08:00
renovate[bot] 7ded6dfe2c chore(config): migrate Renovate config (#2735) 2026-07-08 18:08:04 +08:00
New Future a283f2b52a feat(azure_blob): preserve content type and md5 (#2734) 2026-07-08 11:56:10 +08:00
MadDogOwner bb3bc31609 chore(workflows): upgrade actions (#2728)
upgrade actions cross all workflow files:

- actions/checkout@v6 -> actions/checkout@v7
- actions/cache@v4 -> actions/cache@v6
- actions/download-artifact@v4 -> actions/download-artifact@v8

Signed-off-by: MadDogOwner <xiaoran@xrgzs.top>
2026-07-06 21:41:25 +08:00
Unity_exe 744021d80d chore(wps/driver): remove CompanyID check (#2727)
Fixes #2543
2026-07-06 12:23:12 +08:00
renovate[bot] 94eac2e91e fix(deps): update github.com/t3rm1n4l/go-mega digest to 4de2f12 (#2710) 2026-07-02 07:56:08 +08:00
Yinan Qin a92d59d35c fix(build.sh): remove rm -rf .git/ (#2703)
Signed-off-by: elysia <a.elysia@proton.me>
2026-07-01 17:16:57 +08:00
Yinan Qin fcae009bd3 fix(workflow): using github-script (#2701)
* fix(workflow): using github-script

* fix more

Signed-off-by: Yinan Qin <elysia-best@simplelinux.cn.eu.org>

---------

Signed-off-by: Yinan Qin <elysia-best@simplelinux.cn.eu.org>
2026-07-01 16:45:15 +08:00
Pikachu Ren f63a5e15ee fix(deps): update module github.com/stangelandcl/ppmd v0.1.1 (#2698) 2026-07-01 16:00:18 +08:00
renovate[bot] d85980d3d9 fix(deps): update github.com/cloudsoda/go-smb2 digest to d8c5600 (#2298)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-01 15:41:25 +08:00
renovate[bot] c8306256ff chore(deps): update go toolchain directive to v1.26.4 (#2697)
* chore(deps): update go toolchain directive to v1.26.4

* chore(build.sh): update go version for loongarch abi1.0

Signed-off-by: Yinan Qin <elysia-best@simplelinux.cn.eu.org>

---------

Signed-off-by: Yinan Qin <elysia-best@simplelinux.cn.eu.org>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Yinan Qin <elysia-best@simplelinux.cn.eu.org>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-07-01 15:37:27 +08:00
renovate[bot] 5273a77854 fix(deps): update module github.com/gin-contrib/cors to v1.7.7 (#2688)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-01 15:36:47 +08:00
renovate[bot] cc3d8a06a7 fix(deps): update module github.com/bodgit/sevenzip to v1.6.4 (#2682)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-07-01 15:32:28 +08:00
renovate[bot] 925a2422d6 chore(deps): update github.com/openlistteam/reflink digest to 78760ea (#2693)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Pikachu Ren <40362270+PIKACHUIM@users.noreply.github.com>
2026-07-01 15:29:04 +08:00
renovate[bot] 33015eee9f fix(deps): update module github.com/mholt/archives to v0.1.5 (#2690)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-01 15:28:42 +08:00
renovate[bot] 1be74776f6 fix(deps): update module github.com/ncw/swift/v2 to v2.0.5 (#2691)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-01 15:26:43 +08:00
renovate[bot] 31810f556c fix(deps): update module github.com/sirupsen/logrus to v1.9.4 (#2695)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-01 15:25:27 +08:00
renovate[bot] 30922b6d52 fix(deps): update module github.com/pkg/sftp to v1.13.10 (#2694)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-01 15:24:55 +08:00
renovate[bot] 759adfca21 fix(deps): update module github.com/charmbracelet/bubbles to v0.21.1 (#2686)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-01 13:15:58 +08:00
Pikachu Ren bfbc441fa6 fix(drivers/189pc): sync .torrent file when copy/move/rename/upload (#2522)
* fix(torrent): sync .torrent file when copy/move/rename/upload

* fix(torrent): prevent recursive torrent generation

* fix(189pc): address review comments on torrent sidecar handling

- Only append SHA-1 piece hash when n > 0, preventing spurious piece
  for empty files (BitTorrent v1 requires 0 pieces for 0-byte files)
- Explicitly guard torrent generation with size > 0
- Log findFileByName errors in torrentFollowCopy/Move/Rename instead of
  silently discarding them
- Log WaitBatchTask failure for follow-move operation
- Rename Cloud189File.ParentId to ParentID to match Go initialism
  conventions (consistent with Cloud189Folder.ParentID, Cloud189File.ID)

* revert(ci): go .sum 115 sdk

---------

Co-authored-by: Suyunmeng <Susus0175@proton.me>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-07-01 11:35:45 +08:00
Yinan Qin 866793f485 fix(go.mod): update Go version to 1.25 (#2692)
Signed-off-by: elysia <a.elysia@proton.me>
2026-07-01 11:21:06 +08:00
renovate[bot] 40b39000ab fix(deps): update github.com/t3rm1n4l/go-mega digest to 6845944 (#2088)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-07-01 10:01:24 +08:00
130 changed files with 7882 additions and 11406 deletions
-153
View File
@@ -1,153 +0,0 @@
name: Beta Media Release builds
on:
push:
branches: ["dev-media"]
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: write
jobs:
changelog:
name: Beta Media Release Changelog
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Create or update ref
id: create-or-update-ref
uses: ovsds/create-or-update-ref-action@v1
with:
ref: tags/beta-media
sha: ${{ github.sha }}
- name: Delete beta-media tag
run: git tag -d beta-media
continue-on-error: true
- name: changelog
id: changelog
run: |
git tag -l
npx changelogithub --output CHANGELOG.md
- name: Upload assets to beta-media release
uses: softprops/action-gh-release@v2
with:
body_path: CHANGELOG.md
files: CHANGELOG.md
prerelease: true
tag_name: beta-media
- name: Upload assets to github artifact
uses: actions/upload-artifact@v4
with:
name: beta-media changelog
path: ${{ github.workspace }}/CHANGELOG.md
compression-level: 0
if-no-files-found: error
release:
needs:
- changelog
strategy:
matrix:
include:
- target: "!(*musl*|*windows-arm64*|*windows7-*|*android*|*freebsd*)" # xgo and loongarch
hash: "md5"
- target: "linux-!(arm*)-musl*" #musl-not-arm
hash: "md5-linux-musl"
- target: "linux-arm*-musl*" #musl-arm
hash: "md5-linux-musl-arm"
- target: "windows-arm64" #win-arm64
hash: "md5-windows-arm64"
- target: "windows7-*" #win7
hash: "md5-windows7"
- target: "android-*" #android
hash: "md5-android"
- target: "freebsd-*" #freebsd
hash: "md5-freebsd"
name: Beta Media Release
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.25.0"
- name: Setup web
run: |
frontendRepo="${FRONTEND_REPO:-OpenListTeam/OpenList-Frontend}"
release_json=$(curl -fsSL --max-time 10 \
-H "Authorization: Bearer $GITHUB_TOKEN" \
-H "Accept: application/vnd.github.v3+json" \
"https://api.github.com/repos/$frontendRepo/releases/tags/beta-media")
tar_url=$(echo "$release_json" | jq -r '.assets[].browser_download_url' | grep "openlist-frontend-dist" | grep -v "lite" | grep "\.tar\.gz$")
echo "Downloading frontend from: $tar_url"
curl -fsSL "$tar_url" -o dist.tar.gz
rm -rf public/dist && mkdir -p public/dist
tar -zxvf dist.tar.gz -C public/dist
rm -rf dist.tar.gz
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
FRONTEND_REPO: ${{ vars.FRONTEND_REPO }}
- name: Build
uses: OpenListTeam/cgo-actions@v1.2.2
with:
targets: ${{ matrix.target }}
musl-target-format: $os-$musl-$arch
github-token: ${{ secrets.GITHUB_TOKEN }}
out-dir: build
output: openlist-$target$ext
musl-base-url: "https://github.com/OpenListTeam/musl-compilers/releases/latest/download/"
x-flags: |
github.com/OpenListTeam/OpenList/v4/internal/conf.BuiltAt=$built_at
github.com/OpenListTeam/OpenList/v4/internal/conf.GitAuthor=The OpenList Projects Contributors <noreply@openlist.team>
github.com/OpenListTeam/OpenList/v4/internal/conf.GitCommit=$git_commit
github.com/OpenListTeam/OpenList/v4/internal/conf.Version=$tag
github.com/OpenListTeam/OpenList/v4/internal/conf.WebVersion=rolling
- name: Compress
run: |
bash build.sh zip ${{ matrix.hash }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Upload assets to beta-media release
uses: softprops/action-gh-release@v2
with:
files: build/compress/*
prerelease: true
tag_name: beta-media
- name: Clean illegal characters from matrix.target
id: clean_target_name
run: |
ILLEGAL_CHARS_REGEX='[":<>|*?\\/\r\n]'
CLEANED_TARGET=$(echo "${{ matrix.target }}" | sed -E "s/$ILLEGAL_CHARS_REGEX//g")
echo "Original target: ${{ matrix.target }}"
echo "Cleaned target: $CLEANED_TARGET"
echo "cleaned_target=$CLEANED_TARGET" >> $GITHUB_ENV
- name: Upload assets to github artifact
uses: actions/upload-artifact@v4
with:
name: beta-media builds for ${{ env.cleaned_target }}
path: ${{ github.workspace }}/build/compress/*
compression-level: 0
if-no-files-found: error
+4 -4
View File
@@ -18,7 +18,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
fetch-depth: 0
@@ -108,14 +108,14 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Setup Go
uses: actions/setup-go@v6
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version: "1.25.0"
go-version: "1.26.4"
- name: Setup web
run: bash build.sh dev web
+4 -4
View File
@@ -25,15 +25,15 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
- uses: benjlevesque/short-sha@v3.0
- uses: benjlevesque/short-sha@dbe07338b37c456ce06d23409b35a56a7815eef4 # v4.0
id: short-sha
- name: Setup Go
uses: actions/setup-go@v6
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version: "1.25.0"
go-version: "1.26.4"
- name: Setup web
run: bash build.sh dev web
+1 -1
View File
@@ -14,7 +14,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
fetch-depth: 0
+4 -4
View File
@@ -3,7 +3,7 @@ name: Issue or PR Auto Reply
on:
issues:
types: [opened]
pull_request:
pull_request_target:
types: [opened]
permissions:
@@ -16,7 +16,7 @@ jobs:
if: github.event_name == 'issues'
steps:
- name: Check issue for unchecked tasks and reply
uses: actions/github-script@v7
uses: actions/github-script@v9
with:
script: |
let comment = "";
@@ -78,10 +78,10 @@ jobs:
pr-title-check:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request'
if: github.event_name == 'pull_request_target'
steps:
- name: Check PR title for required prefix and comment
uses: actions/github-script@v7
uses: actions/github-script@v9
with:
script: |
const title = context.payload.pull_request.title || "";
+5 -4
View File
@@ -44,14 +44,15 @@ jobs:
swap-storage: true
- name: Setup Go
uses: actions/setup-go@v6
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version: '1.25.0'
go-version: '1.26.4'
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
fetch-depth: 0
fetch-depth: 1
fetch-tags: true
- name: Install dependencies
if: matrix.target-platform == ''
+24 -24
View File
@@ -43,15 +43,15 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
- uses: actions/setup-go@v6
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version: '1.25.0'
go-version: '1.26'
- name: Cache Musl
id: cache-musl
uses: actions/cache@v4
uses: actions/cache@v6
with:
path: build/musl-libs
key: docker-musl-libs-v2
@@ -83,15 +83,15 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
- uses: actions/setup-go@v6
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version: '1.25.0'
go-version: '1.26.4'
- name: Cache Musl
id: cache-musl
uses: actions/cache@v4
uses: actions/cache@v6
with:
path: build/musl-libs
key: docker-musl-libs-v2
@@ -146,21 +146,21 @@ jobs:
tag_favor: "suffix=-aio,onlatest=true"
steps:
- name: Checkout
uses: actions/checkout@v6
- uses: actions/download-artifact@v4
uses: actions/checkout@v7
- uses: actions/download-artifact@v8
with:
name: ${{ env.ARTIFACT_NAME }}
path: 'build/'
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
- name: Login to GitHub Container Registry
if: env.IMAGE_PUSH == 'true'
uses: docker/login-action@v3
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -168,14 +168,14 @@ jobs:
- name: Login to DockerHub Container Registry
if: env.IMAGE_PUSH == 'true'
uses: docker/login-action@v3
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
with:
username: ${{ vars.DOCKERHUB_ORG_NAME_BACKUP || env.DOCKERHUB_ORG_NAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6
with:
images: |
${{ env.REGISTRY }}/${{ env.GHCR_ORG_NAME }}/${{ env.IMAGE_NAME }}
@@ -190,7 +190,7 @@ jobs:
- name: Build and push
id: docker_build
uses: docker/build-push-action@v6
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
with:
context: .
file: Dockerfile.ci
@@ -230,21 +230,21 @@ jobs:
tag_favor: "suffix=-lite-aio,onlatest=true"
steps:
- name: Checkout
uses: actions/checkout@v6
- uses: actions/download-artifact@v4
uses: actions/checkout@v7
- uses: actions/download-artifact@v8
with:
name: ${{ env.ARTIFACT_NAME_LITE }}
path: 'build/'
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
- name: Login to GitHub Container Registry
if: env.IMAGE_PUSH == 'true'
uses: docker/login-action@v3
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -252,14 +252,14 @@ jobs:
- name: Login to DockerHub Container Registry
if: env.IMAGE_PUSH == 'true'
uses: docker/login-action@v3
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
with:
username: ${{ vars.DOCKERHUB_ORG_NAME_BACKUP || env.DOCKERHUB_ORG_NAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6
with:
images: |
${{ env.REGISTRY }}/${{ env.GHCR_ORG_NAME }}/${{ env.IMAGE_NAME }}
@@ -274,7 +274,7 @@ jobs:
- name: Build and push
id: docker_build
uses: docker/build-push-action@v6
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
with:
context: .
file: Dockerfile.ci
+1 -1
View File
@@ -12,7 +12,7 @@ jobs:
name: Sync GitHub to Gitee
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
with:
fetch-depth: 0
+12 -12
View File
@@ -32,15 +32,15 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v7
- uses: actions/setup-go@v6
- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
with:
go-version: '1.25.0'
go-version: '1.26.4'
- name: Cache Musl
id: cache-musl
uses: actions/cache@v4
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
with:
path: build/musl-libs
key: docker-musl-libs-v2
@@ -97,21 +97,21 @@ jobs:
tag_favor: "suffix=-aio,onlatest=true"
steps:
- name: Checkout
uses: actions/checkout@v6
- uses: actions/download-artifact@v4
uses: actions/checkout@v7
- uses: actions/download-artifact@v8
with:
name: ${{ env.ARTIFACT_NAME }}
path: 'build/'
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
- name: Login to GitHub Container Registry
if: env.IMAGE_PUSH == 'true'
uses: docker/login-action@v3
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -119,14 +119,14 @@ jobs:
- name: Login to DockerHub Container Registry
if: env.IMAGE_PUSH == 'true'
uses: docker/login-action@v3
uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
with:
username: ${{ vars.DOCKERHUB_ORG_NAME_BACKUP || env.DOCKERHUB_ORG_NAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6
with:
images: |
${{ env.REGISTRY }}/${{ env.GHCR_ORG_NAME }}/${{ env.IMAGE_NAME }}
@@ -137,7 +137,7 @@ jobs:
- name: Build and push
id: docker_build
uses: docker/build-push-action@v6
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
with:
context: .
file: Dockerfile.ci
+26 -13
View File
@@ -16,20 +16,33 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Trigger Makefile hash update
uses: peter-evans/repository-dispatch@v3
uses: actions/github-script@v9
with:
token: ${{ secrets.EXTERNAL_REPO_TOKEN_LUCI_APP_OPENLIST }}
repository: ${{ vars.HOOK_REPO || 'OpenListTeam/OpenList-OpenWRT' }}
event-type: update-hashes
client-payload: |
{
"source_repository": "${{ github.repository }}",
"release_tag": "${{ inputs.tag || github.ref_name }}",
"release_name": "${{ inputs.tag || github.ref_name }}",
"release_url": "${{ github.server_url }}/${{ github.repository }}/releases/tag/${{ inputs.tag || github.ref_name }}",
"triggered_by": "${{ github.actor }}",
"trigger_reason": "${{ github.event_name }}"
}
github-token: ${{ secrets.EXTERNAL_REPO_TOKEN_LUCI_APP_OPENLIST }}
script: |
const repository = process.env.REPOSITORY;
const [owner, repo] = repository.split('/');
await github.rest.repos.createDispatchEvent({
owner: owner,
repo: repo,
event_type: 'update-hashes',
client_payload: {
source_repository: process.env.SOURCE_REPOSITORY,
release_tag: process.env.RELEASE_TAG,
release_name: process.env.RELEASE_TAG,
release_url: process.env.RELEASE_URL,
triggered_by: process.env.ACTOR,
trigger_reason: process.env.EVENT_NAME
}
});
env:
REPOSITORY: ${{ vars.HOOK_REPO || 'OpenListTeam/OpenList-OpenWRT' }}
SOURCE_REPOSITORY: ${{ github.repository }}
RELEASE_TAG: ${{ inputs.tag || github.ref_name }}
RELEASE_URL: ${{ github.server_url }}/${{ github.repository }}/releases/tag/${{ inputs.tag || github.ref_name }}
ACTOR: ${{ github.actor }}
EVENT_NAME: ${{ github.event_name }}
- name: Log trigger information
run: |
+4 -2
View File
@@ -22,6 +22,7 @@ output/
# vendor/
/bin/*
*.json
!renovate.json
/build
/data/
/tmp/
@@ -29,6 +30,7 @@ output/
/lang/
/daemon/
/public/dist/*
/!public/dist/README.md
!/public/dist/README.md
.VSCodeCounter
.VSCodeCounter
*.syso
+1 -1
View File
@@ -67,7 +67,7 @@ Thank you for your support and understanding of the OpenList project.
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group, Share)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+1 -1
View File
@@ -50,7 +50,7 @@ OpenList has no official association with third-party derivative projects that u
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group, Share)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+1 -1
View File
@@ -67,7 +67,7 @@ OpenList 是一个由 OpenList 团队独立维护的开源项目,遵循 AGPL-3
- [x] [MediaFire](https://www.mediafire.com)
- [x] [分秒帧](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [和彩云](https://yun.139.com)(个人、家庭、群组)
- [x] [和彩云](https://yun.139.com)(个人、家庭、群组、分享)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [百度网盘](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+1 -1
View File
@@ -50,7 +50,7 @@ OpenList has no official association with third-party derivative projects that u
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group, Share)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+1 -1
View File
@@ -50,7 +50,7 @@ OpenList has no official association with third-party derivative projects that u
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group, Share)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+1 -1
View File
@@ -50,7 +50,7 @@ OpenList has no official association with third-party derivative projects that u
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group, Share)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+1 -1
View File
@@ -66,7 +66,7 @@ OpenListプロジェクトへのご支援とご理解をありがとうござい
- [x] Teambition([中国](https://www.teambition.com), [国際](https://us.teambition.com))
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com)(個人、家族、グループ)
- [x] [139yun](https://yun.139.com)(個人、家族、グループ、共有)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+1 -1
View File
@@ -50,7 +50,7 @@ OpenList has no official association with third-party derivative projects that u
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group, Share)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+1 -1
View File
@@ -67,7 +67,7 @@ Dank u voor uw ondersteuning en begrip
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Persoonlijk, Familie, Groep)
- [x] [139yun](https://yun.139.com) (Persoonlijk, Familie, Groep, Delen)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+1 -1
View File
@@ -50,7 +50,7 @@ OpenList has no official association with third-party derivative projects that u
- [x] [MediaFire](https://www.mediafire.com)
- [x] [Mediatrack](https://www.mediatrack.cn)
- [x] [ProtonDrive](https://proton.me/drive)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group)
- [x] [139yun](https://yun.139.com) (Personal, Family, Group, Share)
- [x] [YandexDisk](https://disk.yandex.com)
- [x] [BaiduNetdisk](http://pan.baidu.com)
- [x] [Terabox](https://www.terabox.com/main)
+3 -9
View File
@@ -178,7 +178,6 @@ BuildWin7() {
}
BuildDev() {
rm -rf .git/
mkdir -p "dist"
muslflags="$(GetMuslStaticLdflags)"
BASE="https://github.com/OpenListTeam/musl-compilers/releases/latest/download/"
@@ -269,7 +268,6 @@ BuildDockerMultiplatform() {
}
BuildRelease() {
rm -rf .git/
mkdir -p "build"
BuildWinArm64 ./build/"$appName"-windows-arm64.exe
BuildWin7 ./build/"$appName"-windows7
@@ -289,7 +287,7 @@ BuildRelease() {
BuildLoongGLIBC() {
local target_abi="$2"
local output_file="$1"
local oldWorldGoVersion="1.25.0"
local oldWorldGoVersion="1.26.4"
local loong_tags
loong_tags=$(GetBuildTagsForTarget "linux-loong64")
@@ -309,13 +307,13 @@ BuildLoongGLIBC() {
# Download and setup patched Go compiler for old-world
if ! curl -fsSL --retry 3 -H "Authorization: Bearer $GITHUB_TOKEN" \
"https://github.com/loong64/loong64-abi1.0-toolchains/releases/download/20250821/go${oldWorldGoVersion}.linux-amd64.tar.gz" \
"https://github.com/loong64-abi1-0/golang/releases/download/go${oldWorldGoVersion}/go${oldWorldGoVersion}.linux-amd64.tar.gz" \
-o go-loong64-abi1.0.tar.gz; then
echo "Error: Failed to download patched Go compiler for old-world ABI1.0"
if [ -n "$GITHUB_TOKEN" ]; then
echo "Error output from curl:"
curl -fsSL --retry 3 -H "Authorization: Bearer $GITHUB_TOKEN" \
"https://github.com/loong64/loong64-abi1.0-toolchains/releases/download/20250821/go${oldWorldGoVersion}.linux-amd64.tar.gz" \
"https://github.com/loong64-abi1-0/golang/releases/download/go${oldWorldGoVersion}/go${oldWorldGoVersion}.linux-amd64.tar.gz" \
-o go-loong64-abi1.0.tar.gz || true
fi
return 1
@@ -442,7 +440,6 @@ BuildLoongGLIBC() {
}
BuildReleaseLinuxMusl() {
rm -rf .git/
mkdir -p "build"
muslflags="$(GetMuslStaticLdflags)"
BASE="https://github.com/OpenListTeam/musl-compilers/releases/latest/download/"
@@ -471,7 +468,6 @@ BuildReleaseLinuxMusl() {
}
BuildReleaseLinuxMuslArm() {
rm -rf .git/
mkdir -p "build"
muslflags="$(GetMuslStaticLdflags)"
BASE="https://github.com/OpenListTeam/musl-compilers/releases/latest/download/"
@@ -502,7 +498,6 @@ BuildReleaseLinuxMuslArm() {
BuildReleaseAndroid() {
rm -rf .git/
mkdir -p "build"
wget https://dl.google.com/android/repository/android-ndk-r26b-linux.zip
unzip android-ndk-r26b-linux.zip
@@ -523,7 +518,6 @@ BuildReleaseAndroid() {
}
BuildReleaseFreeBSD() {
rm -rf .git/
mkdir -p "build/freebsd"
# Get latest FreeBSD 14.x release version from GitHub
+1 -1
View File
@@ -23,7 +23,7 @@ type options struct {
pwd string //de/encrypt password
salt string
filenameEncryption string //reference drivers\crypt\meta.go Addtion
filenameEncryption string //reference drivers\crypt\meta.go Addition
dirnameEncryption string
filenameEncode string
suffix string
-565
View File
@@ -1,565 +0,0 @@
// Command openlist-transcode-worker 是独立的远程转码 Worker 节点。
// 它通过 OpenList 自研协议(HTTP/JSON)注册到 Master,长轮询领任务,调用
// 本机 ffmpeg 转码,并把生成的 HLS 切片实时上传回 Master。
package main
import (
"bytes"
"context"
"encoding/json"
"flag"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"os/signal"
"path/filepath"
"runtime"
"strconv"
"strings"
"sync"
"syscall"
"time"
)
var (
masterURL = flag.String("master", "http://127.0.0.1:5244", "OpenList Master URL")
secret = flag.String("secret", "", "Worker 共享密钥(与 Master 配置一致)")
name = flag.String("name", hostname(), "Worker 名称")
capacity = flag.Int("capacity", 2, "并发任务数")
hwaccel = flag.String("hwaccel", "none", "硬件加速:none/auto/nvenc/qsv/vaapi/amf/videotoolbox")
workdir = flag.String("workdir", "./worker_tmp", "临时工作目录")
ffmpeg = flag.String("ffmpeg", "ffmpeg", "ffmpeg 路径")
)
func hostname() string {
h, _ := os.Hostname()
if h == "" {
h = "worker"
}
return h
}
// ---- 协议结构 ----(与 Master 端 internal/transcode/protocol.go 一致)
type RegisterReq struct {
Name string `json:"name"`
Version string `json:"version"`
Capacity int `json:"capacity"`
HWAccel []string `json:"hwaccel"`
CodecsDecode []string `json:"codecs_decode"`
CodecsEncode []string `json:"codecs_encode"`
MaxResolution string `json:"max_resolution"`
Tags []string `json:"tags"`
}
type RegisterResp struct {
Data struct {
WorkerID string `json:"worker_id"`
HeartbeatInterval int `json:"heartbeat_interval"`
} `json:"data"`
}
type HeartbeatReq struct {
WorkerID string `json:"worker_id"`
Load float64 `json:"load"`
Running []string `json:"running"`
FreeSlots int `json:"free_slots"`
}
type HeartbeatResp struct {
Data struct {
OK bool `json:"ok"`
Kick bool `json:"kick"`
Cancel []string `json:"cancel"`
} `json:"data"`
}
type ClaimReq struct {
WorkerID string `json:"worker_id"`
Slots int `json:"slots"`
Wait int `json:"wait"`
}
type Profile struct {
Name string `json:"name"`
VideoCodec string `json:"video_codec"`
VideoBitrate string `json:"video_bitrate"`
Scale string `json:"scale"`
AudioCodec string `json:"audio_codec"`
AudioBitrate string `json:"audio_bitrate"`
HWAccel string `json:"hwaccel"`
}
type OutputSpec struct {
Format string `json:"format"`
SegmentDuration int `json:"segment_duration"`
}
type Job struct {
ID string `json:"id"`
Path string `json:"path"`
SourceURL string `json:"source_url"`
Profiles []Profile `json:"profiles"`
Output OutputSpec `json:"output"`
CallbackToken string `json:"callback_token"`
Deadline int64 `json:"deadline"`
}
type ClaimResp struct {
Data struct {
Jobs []Job `json:"jobs"`
} `json:"data"`
}
type FinishReq struct {
JobID string `json:"job_id"`
Status string `json:"status"`
Error string `json:"error,omitempty"`
Stats struct {
Elapsed float64 `json:"elapsed"`
BytesOut int64 `json:"bytes_out"`
} `json:"stats"`
}
// ---- HTTP helper ----
var httpClient = &http.Client{Timeout: 60 * time.Second}
func postJSON(path string, body any, out any, bearer string) error {
buf, _ := json.Marshal(body)
req, _ := http.NewRequest(http.MethodPost, *masterURL+path, bytes.NewReader(buf))
req.Header.Set("Content-Type", "application/json")
if bearer != "" {
req.Header.Set("Authorization", "Bearer "+bearer)
}
resp, err := httpClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
raw, _ := io.ReadAll(resp.Body)
return fmt.Errorf("HTTP %d: %s", resp.StatusCode, string(raw))
}
if out != nil {
return json.NewDecoder(resp.Body).Decode(out)
}
return nil
}
// ---- main ----
func main() {
flag.Parse()
if *secret == "" {
fmt.Fprintln(os.Stderr, "missing -secret")
os.Exit(1)
}
if err := os.MkdirAll(*workdir, 0o755); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
workerID, hbInterval, err := registerOnce()
if err != nil {
fmt.Fprintln(os.Stderr, "register failed:", err)
os.Exit(1)
}
fmt.Printf("[worker] registered: id=%s heartbeat=%ds\n", workerID, hbInterval)
ctx, cancel := context.WithCancel(context.Background())
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
go func() {
<-sigCh
fmt.Println("[worker] shutting down")
cancel()
}()
state := &workerState{
id: workerID,
runningMu: sync.Mutex{},
running: map[string]context.CancelFunc{},
}
// 心跳协程
go func() {
t := time.NewTicker(time.Duration(hbInterval) * time.Second)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
state.heartbeat()
}
}
}()
// 工作协程
for i := 0; i < *capacity; i++ {
go state.workLoop(ctx)
}
<-ctx.Done()
time.Sleep(500 * time.Millisecond)
}
func registerOnce() (string, int, error) {
req := RegisterReq{
Name: *name,
Version: "1.0.0",
Capacity: *capacity,
HWAccel: []string{*hwaccel},
CodecsDecode: []string{"h264", "hevc", "av1", "vp9", "mpeg4", "mpeg2video"},
CodecsEncode: []string{"h264", "hevc"},
MaxResolution: "3840x2160",
Tags: []string{runtime.GOOS, runtime.GOARCH},
}
var resp RegisterResp
if err := postJSON("/api/transcode/worker/register", req, &resp, *secret); err != nil {
return "", 0, err
}
if resp.Data.WorkerID == "" {
return "", 0, fmt.Errorf("master returned empty worker_id")
}
if resp.Data.HeartbeatInterval == 0 {
resp.Data.HeartbeatInterval = 10
}
return resp.Data.WorkerID, resp.Data.HeartbeatInterval, nil
}
// ---- 状态机 ----
type workerState struct {
id string
runningMu sync.Mutex
running map[string]context.CancelFunc
}
func (s *workerState) heartbeat() {
s.runningMu.Lock()
ids := make([]string, 0, len(s.running))
for id := range s.running {
ids = append(ids, id)
}
free := *capacity - len(ids)
s.runningMu.Unlock()
var resp HeartbeatResp
if err := postJSON("/api/transcode/worker/heartbeat", HeartbeatReq{
WorkerID: s.id,
Running: ids,
FreeSlots: free,
}, &resp, *secret); err != nil {
fmt.Println("[worker] heartbeat error:", err)
return
}
for _, jid := range resp.Data.Cancel {
s.runningMu.Lock()
if cancel, ok := s.running[jid]; ok {
cancel()
}
s.runningMu.Unlock()
}
}
func (s *workerState) workLoop(ctx context.Context) {
for {
if ctx.Err() != nil {
return
}
var resp ClaimResp
err := postJSON("/api/transcode/worker/claim", ClaimReq{
WorkerID: s.id, Slots: 1, Wait: 20,
}, &resp, *secret)
if err != nil {
fmt.Println("[worker] claim error:", err)
time.Sleep(5 * time.Second)
continue
}
if len(resp.Data.Jobs) == 0 {
continue
}
for _, j := range resp.Data.Jobs {
s.runJob(ctx, j)
}
}
}
func (s *workerState) runJob(parent context.Context, job Job) {
jobCtx, cancel := context.WithCancel(parent)
s.runningMu.Lock()
s.running[job.ID] = cancel
s.runningMu.Unlock()
defer func() {
s.runningMu.Lock()
delete(s.running, job.ID)
s.runningMu.Unlock()
cancel()
}()
startedAt := time.Now()
tmpDir := filepath.Join(*workdir, job.ID)
_ = os.MkdirAll(tmpDir, 0o755)
defer os.RemoveAll(tmpDir)
if len(job.Profiles) == 0 {
s.finish(job, "failed", "no profiles", 0, 0)
return
}
profile := job.Profiles[0]
playlist := filepath.Join(tmpDir, "playlist.m3u8")
segPattern := filepath.Join(tmpDir, "seg-%d.ts")
args := buildFFmpegArgs(job, profile, segPattern, playlist)
cmd := exec.CommandContext(jobCtx, *ffmpeg, args...)
stderr := &bytes.Buffer{}
cmd.Stderr = stderr
fmt.Printf("[worker] running job %s, ffmpeg %s\n", job.ID, strings.Join(args, " "))
if err := cmd.Start(); err != nil {
s.finish(job, "failed", err.Error(), 0, 0)
return
}
// 上传协程
stopUpload := make(chan struct{})
uploadDone := make(chan struct{})
go func() {
defer close(uploadDone)
s.watchAndUpload(jobCtx, job, profile.Name, tmpDir, stopUpload)
}()
err := cmd.Wait()
close(stopUpload)
<-uploadDone
// 最后一次扫描:把还未上传的尾片传上去
s.uploadOnce(job, profile.Name, tmpDir, true)
if err != nil {
if jobCtx.Err() == context.Canceled {
s.finish(job, "cancelled", "cancelled by master", 0, 0)
return
}
s.finish(job, "failed", fmt.Sprintf("%v: %s", err, truncate(stderr.String(), 1024)), 0, 0)
return
}
s.finish(job, "finished", "", time.Since(startedAt).Seconds(), 0)
}
func (s *workerState) finish(job Job, status, errStr string, elapsed float64, bytesOut int64) {
req := FinishReq{
JobID: job.ID, Status: status, Error: errStr,
}
req.Stats.Elapsed = elapsed
req.Stats.BytesOut = bytesOut
if err := postJSON("/api/transcode/worker/job/finish", req, nil, job.CallbackToken); err != nil {
fmt.Printf("[worker] finish report failed: %v\n", err)
}
}
func (s *workerState) watchAndUpload(_ context.Context, job Job, profile, dir string, stop chan struct{}) {
t := time.NewTicker(800 * time.Millisecond)
defer t.Stop()
uploaded := map[int]bool{}
flush := func(final bool) {
s.uploadDelta(job, profile, dir, uploaded, final)
}
for {
select {
case <-stop:
flush(false)
return
case <-t.C:
flush(false)
}
}
}
func (s *workerState) uploadOnce(job Job, profile, dir string, final bool) {
uploaded := map[int]bool{}
s.uploadDelta(job, profile, dir, uploaded, final)
}
// uploadDelta 扫描目录,上传新切片(uploaded 跟踪已传过的 seq;最后一片到 final 时才上传)
func (s *workerState) uploadDelta(job Job, profile, dir string, uploaded map[int]bool, final bool) {
entries, err := os.ReadDir(dir)
if err != nil {
return
}
type segFile struct {
seq int
path string
}
var segs []segFile
for _, e := range entries {
if e.IsDir() {
continue
}
n := e.Name()
if !strings.HasPrefix(n, "seg-") || !strings.HasSuffix(n, ".ts") {
continue
}
num, err := strconv.Atoi(strings.TrimSuffix(strings.TrimPrefix(n, "seg-"), ".ts"))
if err != nil {
continue
}
segs = append(segs, segFile{num, filepath.Join(dir, n)})
}
// 排序
for i := 0; i < len(segs); i++ {
for j := i + 1; j < len(segs); j++ {
if segs[j].seq < segs[i].seq {
segs[i], segs[j] = segs[j], segs[i]
}
}
}
limit := len(segs)
if !final && limit > 0 {
limit-- // 最后一片可能还在写,留到下一轮
}
segDur := job.Output.SegmentDuration
if segDur == 0 {
segDur = 6
}
for i := 0; i < limit; i++ {
seg := segs[i]
if uploaded[seg.seq] {
continue
}
isLast := final && i == limit-1
if err := s.uploadSegment(job, profile, seg.seq, float64(segDur), seg.path, isLast); err != nil {
fmt.Printf("[worker] upload seg-%d failed: %v\n", seg.seq, err)
return
}
uploaded[seg.seq] = true
}
}
func (s *workerState) uploadSegment(job Job, profile string, seq int, dur float64, file string, final bool) error {
f, err := os.Open(file)
if err != nil {
return err
}
defer f.Close()
q := fmt.Sprintf("?job=%s&profile=%s&seq=%d&duration=%f&final=%t",
job.ID, profile, seq, dur, final)
req, _ := http.NewRequest(http.MethodPut, *masterURL+"/api/transcode/worker/segment"+q, f)
req.Header.Set("Content-Type", "video/mp2t")
req.Header.Set("Authorization", "Bearer "+job.CallbackToken)
resp, err := httpClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
raw, _ := io.ReadAll(resp.Body)
return fmt.Errorf("HTTP %d: %s", resp.StatusCode, string(raw))
}
return nil
}
// ---- ffmpeg args(与 master 端 local_runner 保持一致)----
func buildFFmpegArgs(job Job, profile Profile, segPattern, playlist string) []string {
hw := strings.ToLower(profile.HWAccel)
args := []string{"-y", "-loglevel", "warning"}
switch hw {
case "nvenc":
args = append(args, "-hwaccel", "cuda")
case "qsv":
args = append(args, "-hwaccel", "qsv")
case "vaapi":
args = append(args, "-hwaccel", "vaapi", "-hwaccel_output_format", "vaapi", "-vaapi_device", "/dev/dri/renderD128")
case "amf":
args = append(args, "-hwaccel", "d3d11va")
case "videotoolbox":
args = append(args, "-hwaccel", "videotoolbox")
case "auto":
args = append(args, "-hwaccel", "auto")
}
args = append(args, "-i", job.SourceURL)
args = append(args, "-c:v", pickEncoder(profile.VideoCodec, hw))
if profile.VideoBitrate != "" {
args = append(args, "-b:v", profile.VideoBitrate, "-maxrate", profile.VideoBitrate, "-bufsize", profile.VideoBitrate)
}
if profile.Scale != "" {
switch hw {
case "vaapi":
args = append(args, "-vf", "scale_vaapi="+profile.Scale)
case "qsv":
args = append(args, "-vf", "scale_qsv="+profile.Scale)
default:
args = append(args, "-vf", "scale="+profile.Scale)
}
}
if profile.AudioCodec == "copy" {
args = append(args, "-c:a", "copy")
} else {
ac := profile.AudioCodec
if ac == "" {
ac = "aac"
}
args = append(args, "-c:a", ac)
if profile.AudioBitrate != "" {
args = append(args, "-b:a", profile.AudioBitrate)
}
}
segDur := job.Output.SegmentDuration
if segDur == 0 {
segDur = 6
}
args = append(args,
"-f", "hls",
"-hls_time", strconv.Itoa(segDur),
"-hls_playlist_type", "vod",
"-hls_segment_filename", segPattern,
"-hls_flags", "independent_segments",
playlist,
)
return args
}
func pickEncoder(codec, hw string) string {
codec = strings.ToLower(codec)
switch codec {
case "h264", "":
switch hw {
case "nvenc":
return "h264_nvenc"
case "qsv":
return "h264_qsv"
case "vaapi":
return "h264_vaapi"
case "amf":
return "h264_amf"
case "videotoolbox":
return "h264_videotoolbox"
}
return "libx264"
case "hevc", "h265":
switch hw {
case "nvenc":
return "hevc_nvenc"
case "qsv":
return "hevc_qsv"
case "vaapi":
return "hevc_vaapi"
case "amf":
return "hevc_amf"
case "videotoolbox":
return "hevc_videotoolbox"
}
return "libx265"
case "av1":
switch hw {
case "nvenc":
return "av1_nvenc"
case "qsv":
return "av1_qsv"
}
return "libsvtav1"
}
return "libx264"
}
func truncate(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n]
}
-167
View File
@@ -1,167 +0,0 @@
# OpenList 转码协议 v1(自研)
> 用于 OpenList Master 与远程 FFmpeg 转码 Worker 之间的通信。
> 协议本身、所有依赖、推荐编码 (H.264/AV1) 均为开源免费,可商用。
## 总览
```
Player ──► Master(/api/transcode/play) ──► Scheduler.Submit
Worker ──► Master(/api/transcode/worker/register) # 启动一次
Worker ──► Master(/api/transcode/worker/heartbeat) # 周期 10s
Worker ──► Master(/api/transcode/worker/claim) # 长轮询拉任务
Worker ──► Master(/api/transcode/worker/segment) # 流式回推切片
Worker ──► Master(/api/transcode/worker/job/finish) # 上报完成
Player ──► Master(/tc/{job}/{token}/master.m3u8) # 拉取播放列表
Player ──► Master(/tc/{job}/{token}/{profile}/seg-N.ts) # 拉取切片
```
## 通用约定
- 所有请求体均为 JSON。
- Worker 与 Master 间通过两类凭据鉴权:
- **shared_secret**(管理端 `transcode_worker_secret` 设置项):用于 register / heartbeat / claim
- **callback_token**(Master 在 job 中下发,一次性):用于 segment / job/finish
- Header 形式:`Authorization: Bearer <token>`
- 返回包统一使用 OpenList 通用响应:`{"code":200,"message":"success","data":{...}}`
## 端点详细
### POST /api/transcode/worker/register
> 鉴权:Bearer = shared_secret
请求:
```json
{
"name": "gpu-node-01",
"version": "1.0.0",
"capacity": 4,
"hwaccel": ["nvenc"],
"codecs_decode": ["h264","hevc","av1","vp9"],
"codecs_encode": ["h264","hevc"],
"max_resolution": "3840x2160",
"tags": ["linux","amd64"]
}
```
响应:
```json
{
"code":200,"message":"success",
"data": {
"worker_id":"wk_xxx",
"heartbeat_interval":10,
"claim_strategy":"pull",
"protocol_version":"v1"
}
}
```
### POST /api/transcode/worker/heartbeat
> 鉴权:Bearer = shared_secret
> 间隔:register 返回的 `heartbeat_interval`;服务端 30s 未收到则剔除 worker
请求:
```json
{ "worker_id":"wk_xxx", "load":0.4, "running":["job_a"], "free_slots":3 }
```
响应:
```json
{ "code":200,"message":"success","data":{"ok":true,"kick":false,"cancel":["job_x"]}}
```
- `kick=true`:服务端要求 Worker 主动退出
- `cancel`:需要 Worker 立刻终止的 job_ids(异步取消通道)
### POST /api/transcode/worker/claim
> 鉴权:Bearer = shared_secret
> 长轮询:当无任务时,服务端最多挂起 `wait` 秒(最大 30)
请求:
```json
{ "worker_id":"wk_xxx", "slots":1, "wait":20 }
```
响应:
```json
{ "code":200,"message":"success","data":{"jobs":[{
"id":"job_xxx",
"path":"/movies/x.mkv",
"source_url":"https://example.com/d/movies/x.mkv?sign=...",
"profiles":[{
"name":"1080p","video_codec":"h264","video_bitrate":"4000k",
"scale":"1920:-2","audio_codec":"aac","audio_bitrate":"160k",
"hwaccel":"nvenc"
}],
"output":{"format":"hls","segment_duration":6},
"callback_token":"tk_xxx",
"deadline":1731601234
}]}}
```
### PUT /api/transcode/worker/segment
> 鉴权:Bearer = callback_token
> Query 参数:
> - `job` 任务 ID
> - `profile` profile 名(如 `1080p`)
> - `seq` 切片序号(从 0 开始)
> - `duration` 该切片秒数
> - `final` 是否为最后一片(true/false)
>
> Body:原始 MPEG-TS 二进制(`Content-Type: video/mp2t`)
响应:`204 No Content`
### POST /api/transcode/worker/job/finish
> 鉴权:Bearer = callback_token
请求:
```json
{
"job_id":"job_xxx",
"status":"finished",
"error":"",
"stats":{"elapsed":312.4,"fps":120,"speed":2.1,"bytes_out":890123456}
}
```
status 取值:`finished` / `failed` / `cancelled`
## 播放端
### POST /api/fs/transcode/play
> 由前端播放器调用,用户登录态可见
> 请求:`{"path":"/movies/x.mkv"}`
>
> 响应(无需转码):
> `{"transcode":false,"reason":"size below threshold"}`
>
> 响应(已下发转码任务):
> `{"transcode":true,"job_id":"job_xx","master_url":"https://.../tc/.../master.m3u8","profile":"1080p"}`
### GET /tc/:job/:token/master.m3u8
返回多档位主 m3u8。
### GET /tc/:job/:token/:profile/playlist.m3u8
阻塞等待首切片就绪(最多 30s)后返回 HLS 播放列表,未结束时不带 `#EXT-X-ENDLIST`。
### GET /tc/:job/:token/:profile/seg-N.ts
返回 N 号切片;如果尚未生成,最多等待 60s。
## 启动远程 Worker
```bash
# 用 Master 配置中的 transcode_worker_secret 作为密钥
go build -o openlist-worker ./cmd/worker
./openlist-worker \
-master http://openlist.example.com:5244 \
-secret <transcode_worker_secret> \
-capacity 2 \
-hwaccel nvenc \
-ffmpeg /usr/bin/ffmpeg \
-workdir /tmp/openlist-worker
```
## License 说明
- 协议:自研,可自由使用
- FFmpeg:使用系统包提供的 LGPL 版本即可(含 libx264/libx265,libx264 是 GPL,建议在 Worker 镜像中遵循 GPL 公开/动态链接合规)
- 推荐编码:H.264 互联网分发免授权 / AV1 永久免版税
-4
View File
@@ -5,7 +5,6 @@ import (
"strings"
"sync"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
streamPkg "github.com/OpenListTeam/OpenList/v4/internal/stream"
@@ -69,9 +68,6 @@ func (d *Pan115) Link(ctx context.Context, file model.Obj, args model.LinkArgs)
return nil, err
}
userAgent := args.Header.Get("User-Agent")
if userAgent == "" {
userAgent = base.UserAgent
}
downloadInfo, err := d.client.DownloadWithUA(file.(*FileObj).PickCode, userAgent)
if err != nil {
return nil, err
+45 -3
View File
@@ -172,17 +172,59 @@ func (d *Open115) Get(ctx context.Context, path string) (model.Obj, error) {
resp, err := d.client.GetFolderInfoByPath(ctx, path)
if err != nil {
if errors.Is(err, sdk.ErrObjectNotFound) {
return nil, errs.ObjectNotFound
return d.getFromParent(ctx, path, "")
}
return nil, err
}
return &Obj{
obj := &Obj{
Fid: resp.FileID,
Fn: resp.FileName,
Fc: resp.FileCategory,
Sha1: resp.Sha1,
Pc: resp.PickCode,
}, nil
FS: resp.SizeByte,
Upt: parseTime(resp.UTime),
UpPt: parseTime(resp.PTime),
}
if !obj.IsDir() && obj.ModTime().Unix() <= 0 {
return d.getFromParent(ctx, path, obj.GetID())
}
return obj, nil
}
func (d *Open115) getFromParent(ctx context.Context, path, id string) (model.Obj, error) {
path = stdpath.Clean(path)
parent, name := stdpath.Split(path)
parent = stdpath.Clean(parent)
parentID := d.GetRootId()
if stdpath.Clean(parent) != stdpath.Clean(d.parentPath) {
if err := d.WaitLimit(ctx); err != nil {
return nil, err
}
parentInfo, err := d.client.GetFolderInfoByPath(ctx, parent)
if err != nil {
if !errors.Is(err, sdk.ErrObjectNotFound) {
return nil, err
}
parentObj, err := d.getFromParent(ctx, parent, "")
if err != nil {
return nil, err
}
parentID = parentObj.GetID()
} else {
parentID = parentInfo.FileID
}
}
files, err := d.List(ctx, &Obj{Fid: parentID, Fc: "0"}, model.ListArgs{})
if err != nil {
return nil, err
}
for _, file := range files {
if (id != "" && file.GetID() == id) || (id == "" && file.GetName() == name) {
return file, nil
}
}
return nil, errs.ObjectNotFound
}
func (d *Open115) MakeDir(ctx context.Context, parentDir model.Obj, dirName string) (model.Obj, error) {
+9 -1
View File
@@ -1,6 +1,9 @@
package _115_open
import "encoding/json"
import (
"encoding/json"
"strconv"
)
func ParseInt64(v json.Number) (int64, error) {
i, err := v.Int64()
@@ -13,3 +16,8 @@ func ParseInt64(v json.Number) (int64, error) {
}
return int64(0), err
}
func parseTime(value string) int64 {
timestamp, _ := strconv.ParseInt(value, 10, 64)
return timestamp
}
-3
View File
@@ -90,9 +90,6 @@ func (d *Pan115Share) Link(ctx context.Context, file model.Obj, args model.LinkA
if args.Header != nil {
ua = args.Header.Get("User-Agent")
}
if ua == "" {
ua = base.UserAgent
}
downloadInfo, err := d.client.DownloadByShareCodeWithUA(ua, d.ShareCode, d.ReceiveCode, file.GetID())
if err != nil {
return nil, err
+246 -98
View File
@@ -19,6 +19,7 @@ import (
"github.com/OpenListTeam/OpenList/v4/pkg/cron"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/OpenList/v4/pkg/utils/random"
"github.com/avast/retry-go"
log "github.com/sirupsen/logrus"
)
@@ -29,7 +30,9 @@ type Yun139 struct {
Account string
ref *Yun139
PersonalCloudHost string
RootPath string
FamilyCloudHost string
GroupCloudHost string
ProviderRoot string
}
func (d *Yun139) Config() driver.Config {
@@ -42,7 +45,7 @@ func (d *Yun139) GetAddition() driver.Additional {
func (d *Yun139) Init(ctx context.Context) error {
if d.ref == nil {
if len(d.Authorization) == 0 {
if len(d.Authorization) == 0 && !d.isShare() {
if d.Username != "" && d.Password != "" {
log.Infof("139yun: authorization is empty, trying to login with password.")
newAuth, err := d.loginWithPassword()
@@ -54,41 +57,55 @@ func (d *Yun139) Init(ctx context.Context) error {
return fmt.Errorf("authorization is empty and username/password is not provided")
}
}
err := d.refreshToken()
if err != nil {
return err
}
// Query Route Policy
var resp QueryRoutePolicyResp
_, err = d.requestRoute(base.Json{
"userInfo": base.Json{
"userType": 1,
"accountType": 1,
"accountName": d.Account,
},
"modAddrType": 1,
}, &resp)
if err != nil {
return err
}
for _, policyItem := range resp.Data.RoutePolicyList {
if policyItem.ModName == "personal" {
d.PersonalCloudHost = policyItem.HttpsUrl
break
}
}
if len(d.PersonalCloudHost) == 0 {
return fmt.Errorf("PersonalCloudHost is empty")
}
d.cron = cron.NewCron(time.Hour * 12)
d.cron.Do(func() {
if d.Authorization != "" {
err := d.refreshToken()
if err != nil {
log.Errorf("%+v", err)
return err
}
})
// Query Route Policy
var resp QueryRoutePolicyResp
_, err = d.requestRoute(base.Json{
"userInfo": base.Json{
"userType": 1,
"accountType": 1,
"accountName": d.Account,
},
"modAddrType": 1,
}, &resp)
if err != nil {
return err
}
for _, policyItem := range resp.Data.RoutePolicyList {
switch policyItem.ModName {
case "personal":
d.PersonalCloudHost = policyItem.HttpsUrl
case "group":
d.GroupCloudHost = policyItem.HttpsUrl
case "family":
d.FamilyCloudHost = policyItem.HttpsUrl
}
}
if len(d.PersonalCloudHost) == 0 {
return fmt.Errorf("PersonalCloudHost is empty")
}
if d.isGroup() || d.isFamily() {
if len(d.GroupCloudHost) == 0 {
return fmt.Errorf("GroupCloudHost is empty")
}
if len(d.FamilyCloudHost) == 0 {
return fmt.Errorf("FamilyCloudHost is empty")
}
}
d.cron = cron.NewCron(time.Hour * 12)
d.cron.Do(func() {
err := d.refreshToken()
if err != nil {
log.Errorf("%+v", err)
}
})
}
}
switch d.Addition.Type {
case MetaPersonalNew:
@@ -107,15 +124,25 @@ func (d *Yun139) Init(ctx context.Context) error {
if err != nil {
return err
}
case MetaFamily:
case MetaShare:
if len(d.Addition.RootFolderID) == 0 {
// Attempt to obtain data.path as the root via a query and persist it.
if root, err := d.getFamilyRootPath(d.CloudID); err == nil && root != "" {
d.RootFolderID = root
op.MustSaveDriverStorage(d)
}
d.RootFolderID = "root"
}
_, err := d.familyGetFiles(d.RootFolderID)
if len(d.shareEntries()) == 0 {
return fmt.Errorf("link_id is empty")
}
case MetaFamily:
// Attempt to obtain data.path as the root via a query and persist it.
root, err := d.getFamilyRootPath(d.CloudID)
if err != nil || root == "" {
return fmt.Errorf("failed to get family root path: %w", err)
}
d.ProviderRoot = root
if len(d.Addition.RootFolderID) == 0 {
d.RootFolderID = root
op.MustSaveDriverStorage(d)
}
_, err = d.familyGetFiles(d.RootFolderID)
if err != nil {
return err
}
@@ -134,6 +161,19 @@ func (d *Yun139) InitReference(storage driver.Driver) error {
return errs.NotSupport
}
func (d *Yun139) Get(ctx context.Context, path string) (model.Obj, error) {
if !d.isShare() {
return nil, errs.NotImplement
}
if path == "/" {
return &model.Object{ID: "root", Name: "root", IsFolder: true, Path: "/"}, nil
}
if obj, err := d.shareGetObj(path); err == nil {
return obj, nil
}
return nil, errs.ObjectNotFound
}
func (d *Yun139) Drop(ctx context.Context) error {
if d.cron != nil {
d.cron.Stop()
@@ -152,12 +192,23 @@ func (d *Yun139) List(ctx context.Context, dir model.Obj, args model.ListArgs) (
return d.familyGetFiles(dir.GetID())
case MetaGroup:
return d.groupGetFiles(dir.GetID())
case MetaShare:
if dir.GetID() == "root" {
return d.shareGetMergedFiles(d.shareRootEntries())
}
if refs, ok := decodeShareRefs(dir.GetID()); ok {
return d.shareGetMergedFiles(refs)
}
return d.shareGetFilesWithRef(shareRef{LinkID: d.LinkID, NodeID: dir.GetID()}, dir.GetID())
default:
return nil, errs.NotImplement
}
}
func (d *Yun139) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
if file.IsDir() {
return nil, errs.NotFile
}
var url string
var err error
switch d.Addition.Type {
@@ -169,6 +220,16 @@ func (d *Yun139) Link(ctx context.Context, file model.Obj, args model.LinkArgs)
url, err = d.familyGetLink(file.GetID(), file.GetPath())
case MetaGroup:
url, err = d.groupGetLink(file.GetID(), file.GetPath())
case MetaShare:
if refs, ok := decodeShareRefs(file.GetID()); ok && len(refs) > 0 {
return d.shareGetLinkWithRef(refs[0], refs[0].NodeID, args.Type)
}
fallbackRef := shareRef{LinkID: d.LinkID, NodeID: "root"}
if entries := d.shareEntries(); len(entries) > 0 {
fallbackRef.LinkID = entries[0].LinkID
fallbackRef.Password = entries[0].Password
}
return d.shareGetLinkWithRef(fallbackRef, file.GetID(), args.Type)
default:
return nil, errs.NotImplement
}
@@ -179,6 +240,9 @@ func (d *Yun139) Link(ctx context.Context, file model.Obj, args model.LinkArgs)
}
func (d *Yun139) MakeDir(ctx context.Context, parentDir model.Obj, dirName string) error {
if d.isShare() {
return errs.NotImplement
}
var err error
switch d.Addition.Type {
case MetaPersonalNew:
@@ -212,7 +276,7 @@ func (d *Yun139) MakeDir(ctx context.Context, parentDir model.Obj, dirName strin
"accountType": 1,
},
"docLibName": dirName,
"path": path.Join(parentDir.GetPath(), parentDir.GetID()),
"path": d.dirPath(parentDir),
}
pathname := "/orchestration/familyCloud-rebuild/cloudCatalog/v1.0/createCloudDoc"
_, err = d.post(pathname, data, nil)
@@ -225,7 +289,7 @@ func (d *Yun139) MakeDir(ctx context.Context, parentDir model.Obj, dirName strin
},
"groupID": d.CloudID,
"parentFileId": parentDir.GetID(),
"path": path.Join(parentDir.GetPath(), parentDir.GetID()),
"path": d.dirPath(parentDir),
}
pathname := "/orchestration/group-rebuild/catalog/v1.0/createGroupCatalog"
_, err = d.post(pathname, data, nil)
@@ -236,6 +300,9 @@ func (d *Yun139) MakeDir(ctx context.Context, parentDir model.Obj, dirName strin
}
func (d *Yun139) Move(ctx context.Context, srcObj, dstDir model.Obj) (model.Obj, error) {
if d.isShare() {
return nil, errs.NotImplement
}
switch d.Addition.Type {
case MetaPersonalNew:
data := base.Json{
@@ -310,9 +377,9 @@ func (d *Yun139) Move(ctx context.Context, srcObj, dstDir model.Obj) (model.Obj,
var contentList []string
var catalogList []string
if srcObj.IsDir() {
catalogList = append(catalogList, path.Join(srcObj.GetPath(), srcObj.GetID()))
catalogList = append(catalogList, d.dirPath(srcObj))
} else {
contentList = append(contentList, path.Join(srcObj.GetPath(), srcObj.GetID()))
contentList = append(contentList, d.dirPath(srcObj))
}
body := base.Json{
@@ -324,7 +391,7 @@ func (d *Yun139) Move(ctx context.Context, srcObj, dstDir model.Obj) (model.Obj,
"contentList": contentList,
"destCatalogID": dstDir.GetID(),
"destGroupID": d.CloudID,
"destPath": path.Join(dstDir.GetPath(), dstDir.GetID()),
"destPath": d.dirPath(dstDir),
"destType": 0,
"srcGroupID": d.CloudID,
"srcType": 0,
@@ -347,6 +414,9 @@ func (d *Yun139) Move(ctx context.Context, srcObj, dstDir model.Obj) (model.Obj,
}
func (d *Yun139) Rename(ctx context.Context, srcObj model.Obj, newName string) error {
if d.isShare() {
return errs.NotImplement
}
var err error
switch d.Addition.Type {
case MetaPersonalNew:
@@ -425,7 +495,7 @@ func (d *Yun139) Rename(ctx context.Context, srcObj model.Obj, newName string) e
},
"docLibName": newName,
"docLibraryID": srcObj.GetID(),
"path": path.Join(srcObj.GetPath(), srcObj.GetID()),
"path": d.dirPath(srcObj),
}
var resp ModifyCloudDocV2Resp
_, err = d.andAlbumRequest(pathname, data, &resp)
@@ -456,6 +526,9 @@ func (d *Yun139) Rename(ctx context.Context, srcObj model.Obj, newName string) e
}
func (d *Yun139) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
if d.isShare() {
return errs.NotImplement
}
var err error
switch d.Addition.Type {
case MetaPersonalNew:
@@ -525,6 +598,9 @@ func (d *Yun139) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
}
func (d *Yun139) Remove(ctx context.Context, obj model.Obj) error {
if d.isShare() {
return errs.NotImplement
}
switch d.Addition.Type {
case MetaPersonalNew:
data := base.Json{
@@ -615,8 +691,25 @@ func (d *Yun139) getPartSize(size int64) int64 {
}
func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStreamer, up driver.UpdateProgress) error {
switch d.Addition.Type {
case MetaPersonalNew:
if d.isShare() {
return errs.NotImplement
}
// PersonalNew 以及 Group/Family 在非旧流模式时走新上传路径
if d.Addition.Type == MetaPersonalNew ||
((d.isGroup() || d.isFamily()) && !d.UseOldStreamUpload) {
var createPath, getUploadUrlPath, completePath string
if d.isGroup() || d.isFamily() {
// 家庭云和共享群共用同一套新上传 API
createPath = "/dynamic/file/create"
getUploadUrlPath = "/dynamic/file/getUploadUrl"
completePath = "/dynamic/file/complete"
} else {
// MetaPersonalNew
createPath = "/file/create"
getUploadUrlPath = "/file/getUploadUrl"
completePath = "/file/complete"
}
var err error
fullHash := stream.GetHash().GetHash(utils.SHA256)
if len(fullHash) != utils.SHA256.Width {
@@ -671,9 +764,22 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
"type": "file",
"fileRenameMode": "auto_rename",
}
pathname := "/file/create"
// 家庭云和共享群需要额外的参数
if d.isGroup() || d.isFamily() {
if d.CloudID == "" {
return fmt.Errorf("cloud_id is required for group/family upload")
}
data["groupId"] = d.CloudID
if d.isGroup() {
data["groupType"] = 2
} else if d.isFamily() {
data["groupType"] = 1
}
data["catalogType"] = 3
data["seqNo"] = random.String(32)
}
var resp PersonalUploadResp
_, err = d.personalPost(pathname, data, &resp)
_, err = d.newPost(createPath, data, &resp)
if err != nil {
return err
}
@@ -690,10 +796,19 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
if resp.Data.PartInfos != nil {
// Progress
p := driver.NewProgress(size, up)
rateLimited := driver.NewLimitedUploadStream(ctx, stream)
ss, err := streamPkg.NewStreamSectionReader(&streamPkg.FileStream{
Ctx: ctx,
Reader: rateLimited,
Obj: &model.Object{Size: size},
}, int(partSize), &up)
if err != nil {
return err
}
// 先上传前100个分片
err = d.uploadPersonalParts(ctx, partInfos, resp.Data.PartInfos, rateLimited, p)
err = d.uploadPersonalParts(ctx, partInfos, resp.Data.PartInfos, ss, p)
if err != nil {
return err
}
@@ -711,13 +826,12 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
"accountType": 1,
},
}
pathname := "/file/getUploadUrl"
var moreresp PersonalUploadUrlResp
_, err = d.personalPost(pathname, moredata, &moreresp)
_, err = d.newPost(getUploadUrlPath, moredata, &moreresp)
if err != nil {
return err
}
err = d.uploadPersonalParts(ctx, partInfos, moreresp.Data.PartInfos, rateLimited, p)
err = d.uploadPersonalParts(ctx, partInfos, moreresp.Data.PartInfos, ss, p)
if err != nil {
return err
}
@@ -730,7 +844,11 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
"fileId": resp.Data.FileId,
"uploadId": resp.Data.UploadId,
}
_, err = d.personalPost("/file/complete", data, nil)
// 家庭云和共享群需要额外的参数
if d.isGroup() || d.isFamily() {
data["groupId"] = d.CloudID
}
_, err = d.newPost(completePath, data, nil)
if err != nil {
return err
}
@@ -775,11 +893,11 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
}
}
return nil
case MetaPersonal:
fallthrough
case MetaGroup:
fallthrough
case MetaFamily:
}
// 旧上传路径
switch d.Addition.Type {
case MetaPersonal, MetaGroup, MetaFamily:
// 处理冲突
// 获取文件列表
files, err := d.List(ctx, dstDir, model.ListArgs{})
@@ -826,11 +944,11 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
},
}
pathname := "/orchestration/personalCloud/uploadAndDownload/v1.0/pcUploadFileRequest"
if d.isFamily() || d.Addition.Type == MetaGroup {
uploadPath := path.Join(dstDir.GetPath(), dstDir.GetID())
// if dstDir is root folder
if dstDir.GetID() == d.RootFolderID {
uploadPath = d.RootPath
if d.isFamily() || d.isGroup() {
uploadPath := d.dirPath(dstDir)
// 共享群的根目录上传路径为 0
if d.isGroup() && dstDir.GetID() == d.RootFolderID {
uploadPath = "0"
}
data = d.newJson(base.Json{
"fileCount": 1,
@@ -858,56 +976,86 @@ func (d *Yun139) Put(ctx context.Context, dstDir model.Obj, stream model.FileStr
}
size := stream.GetSize()
partSize := d.getPartSize(size)
// Progress
p := driver.NewProgress(size, up)
partSize := d.getPartSize(size)
rateLimited := driver.NewLimitedUploadStream(ctx, stream)
// StreamSectionReader for per-chunk buffering and retry
ss, err := streamPkg.NewStreamSectionReader(&streamPkg.FileStream{
Ctx: ctx,
Reader: rateLimited,
Obj: &model.Object{Size: size},
}, int(partSize), &up)
if err != nil {
return err
}
part := int64(1)
if size > partSize {
part = (size + partSize - 1) / partSize
}
rateLimited := driver.NewLimitedUploadStream(ctx, stream)
for i := int64(0); i < part; i++ {
if utils.IsCanceled(ctx) {
return ctx.Err()
}
start := i * partSize
byteSize := min(size-start, partSize)
limitReader := io.LimitReader(rateLimited, byteSize)
// Update Progress
r := io.TeeReader(limitReader, p)
req, err := http.NewRequestWithContext(ctx, http.MethodPost, resp.Data.UploadResult.RedirectionURL, r)
if err != nil {
return err
rd, getErr := ss.GetSectionReader(start, byteSize)
if getErr != nil {
return getErr
}
req.Header.Set("Content-Type", "text/plain;name="+unicode(stream.GetName()))
req.Header.Set("contentSize", strconv.FormatInt(size, 10))
req.Header.Set("range", fmt.Sprintf("bytes=%d-%d", start, start+byteSize-1))
req.Header.Set("uploadtaskID", resp.Data.UploadResult.UploadTaskID)
req.Header.Set("rangeType", "0")
req.ContentLength = byteSize
res, err := base.HttpClient.Do(req)
err = retry.Do(
func() error {
if _, err := rd.Seek(0, io.SeekStart); err != nil {
return err
}
req, reqErr := http.NewRequestWithContext(ctx, http.MethodPost, resp.Data.UploadResult.RedirectionURL,
io.TeeReader(rd, p))
if reqErr != nil {
return reqErr
}
req.Header.Set("Content-Type", "text/plain;name="+unicode(stream.GetName()))
req.Header.Set("contentSize", strconv.FormatInt(size, 10))
req.Header.Set("range", fmt.Sprintf("bytes=%d-%d", start, start+byteSize-1))
req.Header.Set("uploadtaskID", resp.Data.UploadResult.UploadTaskID)
req.Header.Set("rangeType", "0")
req.ContentLength = byteSize
res, doErr := base.HttpClient.Do(req)
if doErr != nil {
return doErr
}
defer res.Body.Close()
bodyBytes, readErr := io.ReadAll(res.Body)
if readErr != nil {
return fmt.Errorf("error reading response body: %v", readErr)
}
if res.StatusCode != http.StatusOK {
return fmt.Errorf("unexpected status code: %d, body: %s", res.StatusCode, string(bodyBytes))
}
var result InterLayerUploadResult
xmlErr := xml.Unmarshal(bodyBytes, &result)
if xmlErr != nil {
return fmt.Errorf("error parsing XML: %v", xmlErr)
}
if result.ResultCode != 0 {
return fmt.Errorf("upload failed with result code: %d, message: %s", result.ResultCode, result.Msg)
}
return nil
},
retry.Context(ctx),
retry.Attempts(3),
retry.DelayType(retry.BackOffDelay),
retry.Delay(time.Second),
)
ss.FreeSectionReader(rd)
if err != nil {
return err
}
if res.StatusCode != http.StatusOK {
res.Body.Close()
return fmt.Errorf("unexpected status code: %d", res.StatusCode)
}
bodyBytes, err := io.ReadAll(res.Body)
if err != nil {
return fmt.Errorf("error reading response body: %v", err)
}
var result InterLayerUploadResult
err = xml.Unmarshal(bodyBytes, &result)
if err != nil {
return fmt.Errorf("error parsing XML: %v", err)
}
if result.ResultCode != 0 {
return fmt.Errorf("upload failed with result code: %d, message: %s", result.ResultCode, result.Msg)
}
}
return nil
default:
+3 -1
View File
@@ -12,12 +12,14 @@ type Addition struct {
Password string `json:"password" required:"true" secret:"true"`
MailCookies string `json:"mail_cookies" required:"true" type:"text" help:"Cookies from mail.139.com used for login authentication."`
driver.RootID
Type string `json:"type" type:"select" options:"personal_new,family,group,personal" default:"personal_new"`
Type string `json:"type" type:"select" options:"personal_new,family,group,personal,share" default:"personal_new"`
LinkID string `json:"link_id" type:"text" help:"Multiple shares are separated by commas or new lines. Use link_id#password for password-protected shares."`
CloudID string `json:"cloud_id"`
UserDomainID string `json:"user_domain_id" help:"ud_id in Cookie, fill in to show disk usage"`
CustomUploadPartSize int64 `json:"custom_upload_part_size" type:"number" default:"0" help:"0 for auto"`
ReportRealSize bool `json:"report_real_size" type:"bool" default:"true" help:"Enable to report the real file size during upload"`
UseLargeThumbnail bool `json:"use_large_thumbnail" type:"bool" default:"false" help:"Enable to use large thumbnail for images"`
UseOldStreamUpload bool `json:"use_old_stream_upload" type:"bool" default:"false" help:"Enable to use old stream upload method (not support rapid upload)"`
}
var config = driver.Config{
+35
View File
@@ -0,0 +1,35 @@
package _139
import (
"testing"
)
func TestShareEntriesAndRefEncoding(t *testing.T) {
d := &Yun139{Addition: Addition{
Type: MetaShare,
LinkID: "share-a,share-b,share-c#pass",
}}
entries := d.shareEntries()
if len(entries) != 3 {
t.Fatalf("expected 3 share entries, got %d", len(entries))
}
if entries[2].LinkID != "share-c" || entries[2].Password != "pass" {
t.Fatalf("unexpected password share entry: %+v", entries[2])
}
refs := []shareRef{
{LinkID: entries[0].LinkID, Password: entries[0].Password, NodeID: "root-a"},
{LinkID: entries[2].LinkID, Password: entries[2].Password, NodeID: "root-b"},
}
encoded := encodeShareRefs(refs)
decoded, ok := decodeShareRefs(encoded)
if !ok || len(decoded) != len(refs) {
t.Fatalf("failed to decode merged share refs: %q", encoded)
}
for i := range refs {
if decoded[i] != refs[i] {
t.Fatalf("unexpected decoded ref at %d: %+v", i, decoded[i])
}
}
}
+51
View File
@@ -9,6 +9,7 @@ const (
MetaFamily string = "family"
MetaGroup string = "group"
MetaPersonalNew string = "personal_new"
MetaShare string = "share"
)
type BaseResp struct {
@@ -285,6 +286,56 @@ type PersonalUploadUrlResp struct {
}
}
type ShareCatalog struct {
CaID string `json:"caId"`
CaName string `json:"caName"`
UdTime string `json:"udTime"`
}
type ShareContent struct {
CoID string `json:"coId"`
CoName string `json:"coName"`
CoSize int64 `json:"coSize"`
CoType int `json:"coType"`
UdTime string `json:"udTime"`
CoPath string `json:"coPath"`
PresentURL string `json:"presentURL"`
DownloadURL string `json:"downloadURL"`
}
type ShareListResp struct {
BaseResp
Data struct {
LKName string `json:"lkName"`
Passwd string `json:"password"`
CaLst []ShareCatalog `json:"caLst"`
CoLst []ShareContent `json:"coLst"`
} `json:"data"`
}
type ShareContentInfo struct {
PresentURL string `json:"presentURL"`
DownloadURL string `json:"cdnDownLoadUrl"`
}
type ShareDownloadResp struct {
BaseResp
Data struct {
DownloadURL string `json:"downloadURL"`
RedrURL string `json:"redrUrl"`
ExtInfo struct {
CDNDownloadURL string `json:"cdnDownloadUrl"`
} `json:"extInfo"`
} `json:"data"`
}
type ShareContentInfoResp struct {
BaseResp
Data struct {
ContentInfo ShareContentInfo `json:"contentInfo"`
} `json:"data"`
}
type QueryRoutePolicyResp struct {
Success bool `json:"success"`
Code string `json:"code"`
+426 -38
View File
@@ -23,10 +23,13 @@ import (
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
streamPkg "github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/OpenList/v4/pkg/utils/random"
"github.com/avast/retry-go"
"github.com/go-resty/resty/v2"
jsoniter "github.com/json-iterator/go"
log "github.com/sirupsen/logrus"
@@ -39,7 +42,15 @@ const (
// do others that not defined in Driver interface
func (d *Yun139) isFamily() bool {
return d.Type == "family"
return d.Type == MetaFamily
}
func (d *Yun139) isGroup() bool {
return d.Type == MetaGroup
}
func (d *Yun139) isShare() bool {
return d.Type == MetaShare
}
func encodeURIComponent(str string) string {
@@ -339,15 +350,18 @@ func (d *Yun139) familyGetFiles(catalogID string) ([]model.Obj, error) {
},
"sortDirection": 1,
})
// 传入 catalogID 是文件夹的ID,而不是完整路径
// 当传入catalogID为家庭云根目录时,直接留空
if catalogID == d.ProviderRoot {
data["catalogID"] = ""
}
var resp QueryContentListResp
_, err := d.post("/orchestration/familyCloud-rebuild/content/v1.2/queryContentList", data, &resp)
if err != nil {
return nil, err
}
// 返回的是完整的Path: root:/<UserRootID>/<CatalogID>/.../<CatalogID>
path := resp.Data.Path
if catalogID == d.RootFolderID {
d.RootPath = path
}
for _, catalog := range resp.Data.CloudCatalogList {
f := model.Object{
ID: catalog.CatalogID,
@@ -403,9 +417,6 @@ func (d *Yun139) groupGetFiles(catalogID string) ([]model.Obj, error) {
return nil, err
}
path := resp.Data.GetGroupContentResult.ParentCatalogID
if catalogID == d.RootFolderID {
d.RootPath = path
}
for _, catalog := range resp.Data.GetGroupContentResult.CatalogList {
f := model.Object{
ID: catalog.CatalogID,
@@ -485,14 +496,331 @@ func (d *Yun139) groupGetLink(contentId string, path string) (string, error) {
return jsoniter.Get(res, "data", "downloadURL").ToString(), nil
}
var shareAesKeyHex = hex.EncodeToString([]byte("PVGDwmcvfs1uV3d1"))
func (d *Yun139) shareHeaders() map[string]string {
auth := d.getAuthorization()
if auth != "" && !strings.HasPrefix(strings.ToLower(auth), "basic ") {
auth = "Basic " + auth
}
headers := map[string]string{
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0",
"Accept": "application/json, text/plain, */*",
"Content-Type": "application/json;charset=UTF-8",
"X-Deviceinfo": "||9|12.27.0|firefox|140.0|||linux unknown|1920X526|zh-CN|||",
"hcy-cool-flag": "1",
"CMS-DEVICE": "default",
"x-m4c-caller": "PC",
"X-Yun-Api-Version": "v1",
"Origin": "https://yun.139.com",
"Referer": "https://yun.139.com/",
}
if auth != "" {
headers["Authorization"] = auth
}
return headers
}
func (d *Yun139) sharePost(pathname string, data interface{}, resp interface{}) ([]byte, error) {
url := "https://share-kd-njs.yun.139.com/yun-share" + pathname
return d.yun139EncryptedRequest(url, data, d.shareHeaders(), shareAesKeyHex, resp)
}
type shareRef struct {
LinkID string
Password string
NodeID string
}
const multiShareRefPrefix = "shares:"
func encodeShareRef(linkID, password, nodeID string) string {
return url.PathEscape(linkID) + "|" + url.PathEscape(password) + "|" + url.PathEscape(nodeID)
}
func decodeShareRef(id string) (shareRef, bool) {
parts := strings.SplitN(id, "|", 3)
if len(parts) != 3 {
return shareRef{}, false
}
linkID, err1 := url.PathUnescape(parts[0])
password, err2 := url.PathUnescape(parts[1])
nodeID, err3 := url.PathUnescape(parts[2])
if err1 != nil || err2 != nil || err3 != nil || linkID == "" {
return shareRef{}, false
}
return shareRef{LinkID: linkID, Password: password, NodeID: nodeID}, true
}
func encodeShareRefs(refs []shareRef) string {
if len(refs) == 1 {
ref := refs[0]
return encodeShareRef(ref.LinkID, ref.Password, ref.NodeID)
}
data, err := utils.Json.Marshal(refs)
if err != nil {
return ""
}
return multiShareRefPrefix + base64.RawURLEncoding.EncodeToString(data)
}
func decodeShareRefs(id string) ([]shareRef, bool) {
if !strings.HasPrefix(id, multiShareRefPrefix) {
ref, ok := decodeShareRef(id)
if !ok {
return nil, false
}
return []shareRef{ref}, true
}
data, err := base64.RawURLEncoding.DecodeString(strings.TrimPrefix(id, multiShareRefPrefix))
if err != nil {
return nil, false
}
var refs []shareRef
if err = utils.Json.Unmarshal(data, &refs); err != nil || len(refs) == 0 {
return nil, false
}
return refs, true
}
func (d *Yun139) shareEntries() []struct{ LinkID, Password string } {
raw := strings.TrimSpace(d.LinkID)
if raw == "" {
return nil
}
parts := strings.FieldsFunc(raw, func(r rune) bool {
return r == '\n' || r == '\r' || r == ',' || r == ';'
})
entries := make([]struct{ LinkID, Password string }, 0, len(parts))
for _, part := range parts {
part = strings.TrimSpace(part)
if part == "" {
continue
}
entry := struct{ LinkID, Password string }{LinkID: part}
if linkID, password, ok := strings.Cut(part, "#"); ok {
entry.LinkID = strings.TrimSpace(linkID)
entry.Password = strings.TrimSpace(password)
}
if entry.LinkID != "" {
entries = append(entries, entry)
}
}
return entries
}
func (d *Yun139) shareRootEntries() []shareRef {
entries := d.shareEntries()
refs := make([]shareRef, 0, len(entries))
for _, entry := range entries {
refs = append(refs, shareRef{LinkID: entry.LinkID, Password: entry.Password, NodeID: "root"})
}
return refs
}
func (d *Yun139) shareGetFilesWithRef(ref shareRef, pCaID string) ([]model.Obj, error) {
if ref.NodeID == "" {
ref.NodeID = "root"
}
if pCaID == "" {
pCaID = ref.NodeID
}
data := base.Json{
"getOutLinkInfoReq": base.Json{
"account": d.getAccount(),
"linkID": ref.LinkID,
"passwd": ref.Password,
"pCaID": pCaID,
},
}
var resp ShareListResp
_, err := d.sharePost("/richlifeApp/devapp/IOutLink/getOutLinkInfoV6", data, &resp)
if err != nil {
return nil, err
}
files := make([]model.Obj, 0)
for _, catalog := range resp.Data.CaLst {
modTime, _ := time.ParseInLocation("20060102150405", catalog.UdTime, utils.CNLoc)
f := model.Object{
ID: encodeShareRef(ref.LinkID, ref.Password, catalog.CaID),
Name: catalog.CaName,
Modified: modTime,
IsFolder: true,
}
files = append(files, &f)
}
for _, content := range resp.Data.CoLst {
name := content.CoName
size := content.CoSize
modTime, _ := time.ParseInLocation("20060102150405", content.UdTime, utils.CNLoc)
f := model.Object{
ID: encodeShareRef(ref.LinkID, ref.Password, content.CoID),
Name: name,
Size: size,
Modified: modTime,
}
files = append(files, &f)
}
return files, nil
}
func (d *Yun139) shareGetMergedFiles(refs []shareRef) ([]model.Obj, error) {
files := make([]model.Obj, 0)
indices := make(map[string]int)
var firstErr error
for _, ref := range refs {
items, err := d.shareGetFilesWithRef(ref, ref.NodeID)
if err != nil {
if firstErr == nil {
firstErr = err
}
continue
}
for _, item := range items {
idx, exists := indices[item.GetName()]
if !exists {
indices[item.GetName()] = len(files)
files = append(files, item)
continue
}
if !files[idx].IsDir() || !item.IsDir() {
continue
}
existingRefs, ok1 := decodeShareRefs(files[idx].GetID())
itemRefs, ok2 := decodeShareRefs(item.GetID())
if !ok1 || !ok2 {
continue
}
mergedRefs := append(existingRefs, itemRefs...)
files[idx] = &model.Object{
ID: encodeShareRefs(mergedRefs),
Name: item.GetName(),
Modified: item.ModTime(),
IsFolder: true,
}
}
}
if len(files) == 0 && firstErr != nil {
return nil, firstErr
}
return files, nil
}
func (d *Yun139) shareGetObj(reqPath string) (model.Obj, error) {
reqPath = utils.FixAndCleanPath(reqPath)
if reqPath == "/" {
return &model.Object{ID: "root", Name: "root", IsFolder: true, Path: "/"}, nil
}
parts := strings.Split(strings.Trim(reqPath, "/"), "/")
currentRefs := d.shareRootEntries()
currentPath := ""
var currentObj model.Obj
for idx, part := range parts {
items, err := d.shareGetMergedFiles(currentRefs)
if err != nil {
return nil, err
}
matched := false
for _, item := range items {
if item.GetName() != part {
continue
}
matched = true
currentObj = item
currentPath = path.Join(currentPath, item.GetName())
if item.IsDir() {
itemRefs, ok := decodeShareRefs(item.GetID())
if !ok {
return nil, errs.ObjectNotFound
}
currentRefs = itemRefs
break
}
if idx != len(parts)-1 {
return nil, errs.ObjectNotFound
}
}
if !matched {
return nil, errs.ObjectNotFound
}
}
if currentObj == nil {
return nil, errs.ObjectNotFound
}
if setter, ok := currentObj.(model.SetPath); ok {
setter.SetPath(currentPath)
}
return currentObj, nil
}
func (d *Yun139) shareGetLinkWithRef(ref shareRef, coID string, linkType string) (*model.Link, error) {
data := base.Json{
"getContentInfoFromOutLinkReq": base.Json{
"contentId": coID,
"linkID": ref.LinkID,
"passwd": ref.Password,
"account": d.getAccount(),
},
}
var resp ShareContentInfoResp
body, err := d.sharePost("/richlifeApp/devapp/IOutLink/getContentInfoFromOutLink", data, &resp)
if err != nil {
return nil, err
}
res := resp.Data.ContentInfo
if linkType == "video_preview" || linkType == "preview" || linkType == "thumb" {
if res.PresentURL == "" {
return nil, fmt.Errorf("failed to get preview link")
}
return &model.Link{URL: res.PresentURL}, nil
}
if d.getAccount() == "" {
if res.PresentURL != "" {
return &model.Link{URL: res.PresentURL}, nil
}
return nil, fmt.Errorf("139 share download requires account authentication")
}
downloadReq := base.Json{
"dlFromOutLinkReqV3": base.Json{
"account": d.getAccount(),
"linkID": ref.LinkID,
"passwd": ref.Password,
"coIDLst": base.Json{
"item": []string{coID},
},
},
}
var downloadResp ShareDownloadResp
downloadBody, err := d.sharePost("/richlifeApp/devapp/IOutLink/dlFromOutLinkV3", downloadReq, &downloadResp)
if err != nil {
return nil, err
}
if downloadResp.Data.ExtInfo.CDNDownloadURL != "" {
return &model.Link{URL: downloadResp.Data.ExtInfo.CDNDownloadURL}, nil
}
if downloadResp.Data.RedrURL != "" {
return &model.Link{URL: downloadResp.Data.RedrURL}, nil
}
if downloadResp.Data.DownloadURL != "" {
return &model.Link{URL: downloadResp.Data.DownloadURL}, nil
}
log.Debugf("[139Share] content info without embedded download url: %s", string(body))
log.Debugf("[139Share] download response without direct url: %s", string(downloadBody))
return nil, fmt.Errorf("failed to get link")
}
func unicode(str string) string {
textQuoted := strconv.QuoteToASCII(str)
textUnquoted := textQuoted[1 : len(textQuoted)-1]
return textUnquoted
}
func (d *Yun139) personalRequest(pathname string, method string, callback base.ReqCallback, resp interface{}) ([]byte, error) {
url := d.getPersonalCloudHost() + pathname
func (d *Yun139) newRequest(url string, method string, callback base.ReqCallback, resp interface{}) ([]byte, error) {
req := base.RestyClient.R()
randStr := random.String(16)
ts := time.Now().Format("2006-01-02 15:04:05")
@@ -554,7 +882,21 @@ func (d *Yun139) personalRequest(pathname string, method string, callback base.R
}
func (d *Yun139) personalPost(pathname string, data interface{}, resp interface{}) ([]byte, error) {
return d.personalRequest(pathname, http.MethodPost, func(req *resty.Request) {
return d.newRequest(d.getPersonalCloudHost()+pathname, http.MethodPost, func(req *resty.Request) {
req.SetBody(data)
}, resp)
}
func (d *Yun139) newPost(pathname string, data interface{}, resp interface{}) ([]byte, error) {
var url string
switch d.Type {
case MetaFamily, MetaGroup:
// this is on purpose
url = d.getGroupCloudHost() + pathname
default:
url = d.getPersonalCloudHost() + pathname
}
return d.newRequest(url, http.MethodPost, func(req *resty.Request) {
req.SetBody(data)
}, resp)
}
@@ -681,7 +1023,21 @@ func (d *Yun139) getPersonalCloudHost() string {
return d.PersonalCloudHost
}
func (d *Yun139) uploadPersonalParts(ctx context.Context, partInfos []PartInfo, uploadPartInfos []PersonalPartInfo, rateLimited *driver.RateLimitReader, p *driver.Progress) error {
func (d *Yun139) getFamilyCloudHost() string {
if d.ref != nil {
return d.ref.getFamilyCloudHost()
}
return d.FamilyCloudHost
}
func (d *Yun139) getGroupCloudHost() string {
if d.ref != nil {
return d.ref.getGroupCloudHost()
}
return d.GroupCloudHost
}
func (d *Yun139) uploadPersonalParts(ctx context.Context, partInfos []PartInfo, uploadPartInfos []PersonalPartInfo, ss streamPkg.StreamSectionReader, p *driver.Progress) error {
// 确保数组以 PartNumber 从小到大排序
sort.Slice(uploadPartInfos, func(i, j int) bool {
return uploadPartInfos[i].PartNumber < uploadPartInfos[j].PartNumber
@@ -693,31 +1049,51 @@ func (d *Yun139) uploadPersonalParts(ctx context.Context, partInfos []PartInfo,
return fmt.Errorf("invalid PartNumber %d: index out of bounds (partInfos length: %d)", uploadPartInfo.PartNumber, len(partInfos))
}
partSize := partInfos[index].PartSize
offset := partInfos[index].ParallelHashCtx.PartOffset
log.Debugf("[139] uploading part %+v/%+v", index, len(partInfos))
limitReader := io.LimitReader(rateLimited, partSize)
r := io.TeeReader(limitReader, p)
req, err := http.NewRequestWithContext(ctx, http.MethodPut, uploadPartInfo.UploadUrl, r)
if err != nil {
return err
rd, getErr := ss.GetSectionReader(offset, partSize)
if getErr != nil {
return getErr
}
req.Header.Set("Content-Type", "application/octet-stream")
req.Header.Set("Content-Length", fmt.Sprint(partSize))
req.Header.Set("Origin", "https://yun.139.com")
req.Header.Set("Referer", "https://yun.139.com/")
req.ContentLength = partSize
err = func() error {
res, err := base.HttpClient.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
log.Debugf("[139] uploaded: %+v", res)
if res.StatusCode != http.StatusOK {
body, _ := io.ReadAll(res.Body)
return fmt.Errorf("unexpected status code: %d, body: %s", res.StatusCode, string(body))
}
return nil
}()
// Save progress before this part so retries don't double-count bytes
partDoneStart := p.Done
err := retry.Do(
func() error {
// Reset progress to the start of this part on each attempt
p.Done = partDoneStart
if _, seekErr := rd.Seek(0, io.SeekStart); seekErr != nil {
return seekErr
}
req, reqErr := http.NewRequestWithContext(ctx, http.MethodPut, uploadPartInfo.UploadUrl, io.TeeReader(rd, p))
if reqErr != nil {
return reqErr
}
req.Header.Set("Content-Type", "application/octet-stream")
req.Header.Set("Content-Length", fmt.Sprint(partSize))
req.Header.Set("Origin", "https://yun.139.com")
req.Header.Set("Referer", "https://yun.139.com/")
req.ContentLength = partSize
res, doErr := base.HttpClient.Do(req)
if doErr != nil {
return doErr
}
defer res.Body.Close()
log.Debugf("[139] uploaded: %+v", res)
if res.StatusCode != http.StatusOK {
body, _ := io.ReadAll(res.Body)
return fmt.Errorf("unexpected status code: %d, body: %s", res.StatusCode, string(body))
}
return nil
},
retry.Context(ctx),
retry.Attempts(3),
retry.DelayType(retry.BackOffDelay),
retry.Delay(time.Second),
)
ss.FreeSectionReader(rd)
if err != nil {
return err
}
@@ -881,7 +1257,6 @@ func (d *Yun139) step2_get_single_token(sid string) (string, error) {
}
exchangePassidHeaders := map[string]string{
"Host": "smsrebuild1.mail.10086.cn",
"Cookie": rmkey,
"Content-Type": "text/xml; charset=utf-8",
"Accept-Encoding": "gzip",
@@ -1156,7 +1531,6 @@ func (d *Yun139) step3_third_party_login(dycpwd string) (string, error) {
"x-UserAgent": "android|23116PN5BC|android15|1.2.6|||1440x3200|10246600",
"x-DeviceInfo": "4|127.0.0.1|5|1.2.6|Xiaomi|23116PN5BC||02-00-00-00-00-00|android 15|1440x3200|android|||",
"Content-Type": "text/plain;charset=UTF-8",
"Host": "user-njs.yun.139.com",
"Accept-Encoding": "gzip",
"User-Agent": "okhttp/3.12.2",
}
@@ -1237,10 +1611,9 @@ func (d *Yun139) loginWithPassword() (string, error) {
}
func (d *Yun139) andAlbumRequest(pathname string, body interface{}, resp interface{}) ([]byte, error) {
url := "https://group.yun.139.com/hcy/family/adapter/andAlbum/openApi" + pathname
url := d.getFamilyCloudHost() + "/andAlbum/openApi" + pathname
headers := map[string]string{
"Host": "group.yun.139.com",
"authorization": "Basic " + d.getAuthorization(),
"x-svctype": "2",
"hcy-cool-flag": "1",
@@ -1325,6 +1698,21 @@ func (d *Yun139) getGroupRootByCloudID(cloudID string) (string, error) {
return "", fmt.Errorf("no root found in group response")
}
// dirPath returns the full path for a directory object.
// For family root (Path="" from framework), needs "root:/"+id prefix.
// Non-root objects already have their API path in GetPath() from List responses.
func (d *Yun139) dirPath(dir model.Obj) string {
p := dir.GetPath()
id := dir.GetID()
if p == "" {
if d.isFamily() {
return "root:/" + id
}
return id
}
return path.Join(p, id)
}
// getFamilyRootPath 查询 family 的上层 path(data.path)
// 返回值已去除前缀 "root:/"(或 "root:"),直接返回纯 ID 或 path 部分,便于持久化为 RootFolderID。
func (d *Yun139) getFamilyRootPath(cloudID string) (string, error) {
+30 -1
View File
@@ -262,6 +262,16 @@ func (y *Cloud189PC) Move(ctx context.Context, srcObj, dstDir model.Obj) (model.
if err = y.WaitBatchTask("MOVE", resp.TaskID, time.Millisecond*400); err != nil {
return nil, err
}
// 跟随移动 torrent 文件
if !srcObj.IsDir() {
var srcFolderId string
if f, ok := srcObj.(*Cloud189File); ok {
srcFolderId = f.ParentID
}
y.torrentFollowMove(srcFolderId, srcObj.GetName(), dstDir)
}
return srcObj, nil
}
@@ -298,6 +308,12 @@ func (y *Cloud189PC) Rename(ctx context.Context, srcObj model.Obj, newName strin
}
return nil, err
}
// 跟随重命名 torrent 文件
if f, ok := srcObj.(*Cloud189File); ok {
y.torrentFollowRename(f.ParentID, srcObj.GetName(), newName)
}
switch f := srcObj.(type) {
case *Cloud189File:
return resp.toFile(f), nil
@@ -319,7 +335,20 @@ func (y *Cloud189PC) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
if err != nil {
return err
}
return y.WaitBatchTask("COPY", resp.TaskID, time.Second)
if err = y.WaitBatchTask("COPY", resp.TaskID, time.Second); err != nil {
return err
}
// 跟随复制 torrent 文件
if !srcObj.IsDir() {
var srcFolderId string
if f, ok := srcObj.(*Cloud189File); ok {
srcFolderId = f.ParentID
}
y.torrentFollowCopy(srcFolderId, srcObj.GetName(), dstDir)
}
return nil
}
func (y *Cloud189PC) Remove(ctx context.Context, obj model.Obj) error {
+158
View File
@@ -1,6 +1,7 @@
package _189pc
import (
"bytes"
"context"
"crypto/sha1"
"encoding/hex"
@@ -8,10 +9,13 @@ import (
"io"
"net/url"
"strings"
"time"
"github.com/go-resty/resty/v2"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/torrent"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
)
@@ -162,6 +166,36 @@ func (y *Cloud189PC) RapidUploadFromTorrent(ctx context.Context, dstDir model.Ob
return nil, fmt.Errorf("提交上传失败: %w", err)
}
// 秒传成功后,将 torrent 文件上传到目标目录(异步,不影响秒传结果)
if y.Addition.GenerateTorrent {
capturedDstDir := dstDir
capturedIsFamily := isFamily
go func() {
torrentName := fileName + ".cas.torrent"
infoHash, _ := GetInfoHashHex(torrentData)
utils.Log.Infof("秒传成功,上传 torrent: %s (info_hash: %s, size: %d bytes)",
torrentName, infoHash, len(torrentData))
torrentFileStream := &stream.FileStream{
Ctx: context.Background(),
Obj: &model.Object{
Name: torrentName,
Size: int64(len(torrentData)),
IsFolder: false,
},
Reader: bytes.NewReader(torrentData),
Mimetype: "application/x-bittorrent",
}
_, uploadErr := y.fastUpload(context.Background(), capturedDstDir, torrentFileStream, func(p float64) {}, capturedIsFamily, false, false)
if uploadErr != nil {
utils.Log.Warnf("上传 torrent 文件失败: %v", uploadErr)
} else {
utils.Log.Infof("torrent 文件已上传: %s", torrentName)
op.Cache.DeleteDirectory(y, capturedDstDir.GetPath())
}
}()
}
return resp.toFile(), nil
}
@@ -258,6 +292,10 @@ func GetInfoHashHex(torrentData []byte) (string, error) {
return hex.EncodeToString(t.InfoHash), nil
}
func isCASTorrentFile(fileName string) bool {
return strings.HasSuffix(fileName, ".cas.torrent")
}
// ComputeSliceMD5sFromReader 从 reader 中计算每个 10MB 分片的 MD5
// 返回:整文件 MD5、分片 MD5 列表
func ComputeSliceMD5sFromReader(reader io.Reader, sliceSize int64) (string, []string, error) {
@@ -289,3 +327,123 @@ func ComputeSliceMD5sFromReader(reader io.Reader, sliceSize int64) (string, []st
fileMD5Hex := strings.ToUpper(hex.EncodeToString(fileMD5Hash.Sum(nil)))
return fileMD5Hex, sliceMD5s, nil
}
// torrentFollowCopy 跟随复制 torrent 文件(异步,不影响主操作)
// srcFolderId: 源文件所在目录 ID
// srcFileName: 源文件名
// dstDir: 目标目录
func (y *Cloud189PC) torrentFollowCopy(srcFolderId string, srcFileName string, dstDir model.Obj) {
if !y.Addition.GenerateTorrent {
return
}
if srcFolderId == "" {
return
}
torrentName := srcFileName + ".cas.torrent"
isFamily := y.isFamily()
go func() {
torrentFile, err := y.findFileByName(context.Background(), torrentName, srcFolderId, isFamily)
if err != nil {
utils.Log.Debugf("查找 torrent 文件失败(跟随复制): %v", err)
return
}
// 复制 torrent 文件到目标目录
resp, copyErr := y.CreateBatchTask("COPY", IF(isFamily, y.FamilyID, ""), dstDir.GetID(),
map[string]string{"targetFileName": dstDir.GetName()},
BatchTaskInfo{
FileId: torrentFile.GetID(),
FileName: torrentFile.GetName(),
IsFolder: 0,
})
if copyErr != nil {
utils.Log.Warnf("跟随复制 torrent 文件失败: %v", copyErr)
return
}
if err = y.WaitBatchTask("COPY", resp.TaskID, time.Second); err != nil {
utils.Log.Warnf("等待跟随复制 torrent 文件失败: %v", err)
}
}()
}
// torrentFollowMove 跟随移动 torrent 文件(异步,不影响主操作)
// srcFolderId: 源文件所在目录 ID
// srcFileName: 源文件名
// dstDir: 目标目录
func (y *Cloud189PC) torrentFollowMove(srcFolderId string, srcFileName string, dstDir model.Obj) {
if !y.Addition.GenerateTorrent {
return
}
if srcFolderId == "" {
return
}
torrentName := srcFileName + ".cas.torrent"
isFamily := y.isFamily()
go func() {
torrentFile, err := y.findFileByName(context.Background(), torrentName, srcFolderId, isFamily)
if err != nil {
utils.Log.Debugf("查找 torrent 文件失败(跟随移动): %v", err)
return
}
// 移动 torrent 文件到目标目录
resp, moveErr := y.CreateBatchTask("MOVE", IF(isFamily, y.FamilyID, ""), dstDir.GetID(),
map[string]string{"targetFileName": dstDir.GetName()},
BatchTaskInfo{
FileId: torrentFile.GetID(),
FileName: torrentFile.GetName(),
IsFolder: 0,
})
if moveErr != nil {
utils.Log.Warnf("跟随移动 torrent 文件失败: %v", moveErr)
return
}
if err = y.WaitBatchTask("MOVE", resp.TaskID, time.Millisecond*400); err != nil {
utils.Log.Warnf("等待跟随移动 torrent 文件失败: %v", err)
}
}()
}
// torrentFollowRename 跟随重命名 torrent 文件(异步,不影响主操作)
// folderId: 文件所在目录 ID
// oldFileName: 原文件名
// newFileName: 新文件名
func (y *Cloud189PC) torrentFollowRename(folderId string, oldFileName string, newFileName string) {
if !y.Addition.GenerateTorrent {
return
}
if folderId == "" {
return
}
oldTorrentName := oldFileName + ".cas.torrent"
newTorrentName := newFileName + ".cas.torrent"
isFamily := y.isFamily()
go func() {
torrentFile, err := y.findFileByName(context.Background(), oldTorrentName, folderId, isFamily)
if err != nil {
utils.Log.Debugf("查找 torrent 文件失败(跟随重命名): %v", err)
return
}
// 重命名 torrent 文件
queryParam := make(map[string]string)
fullUrl := API_URL
method := "POST"
if isFamily {
fullUrl += "/family/file"
method = "GET"
queryParam["familyId"] = y.FamilyID
}
fullUrl += "/renameFile.action"
queryParam["fileId"] = torrentFile.GetID()
queryParam["destFileName"] = newTorrentName
_, renameErr := y.request(fullUrl, method, func(req *resty.Request) {
req.SetContext(context.Background()).SetQueryParams(queryParam)
}, nil, &RenameResp{}, isFamily)
if renameErr != nil {
utils.Log.Warnf("跟随重命名 torrent 文件失败: %v", renameErr)
}
}()
}
+5 -4
View File
@@ -166,10 +166,11 @@ type FamilyInfoResp struct {
/*文件部分*/
// 文件
type Cloud189File struct {
ID String `json:"id"`
Name string `json:"name"`
Size int64 `json:"size"`
Md5 string `json:"md5"`
ID String `json:"id"`
Name string `json:"name"`
Size int64 `json:"size"`
Md5 string `json:"md5"`
ParentID string `json:"-"` // 由 getFiles 设置,不从 JSON 解析
LastOpTime Time `json:"lastOpTime"`
CreateDate Time `json:"createDate"`
+63 -1
View File
@@ -210,6 +210,7 @@ func (y *Cloud189PC) getFiles(ctx context.Context, fileId string, isFamily bool)
res = append(res, &resp.FileListAO.FolderList[i])
}
for i := 0; i < len(resp.FileListAO.FileList); i++ {
resp.FileListAO.FileList[i].ParentID = fileId
res = append(res, &resp.FileListAO.FileList[i])
}
}
@@ -910,6 +911,11 @@ func (y *Cloud189PC) RapidUpload(ctx context.Context, dstDir model.Obj, stream m
// 快传
func (y *Cloud189PC) FastUpload(ctx context.Context, dstDir model.Obj, file model.FileStreamer, up driver.UpdateProgress, isFamily bool, overwrite bool) (model.Obj, error) {
generateTorrent := y.Addition.GenerateTorrent && !isCASTorrentFile(file.GetName())
return y.fastUpload(ctx, dstDir, file, up, isFamily, overwrite, generateTorrent)
}
func (y *Cloud189PC) fastUpload(ctx context.Context, dstDir model.Obj, file model.FileStreamer, up driver.UpdateProgress, isFamily bool, overwrite bool, generateTorrent bool) (model.Obj, error) {
var (
cache = file.GetFile()
tmpF *os.File
@@ -947,6 +953,9 @@ func (y *Cloud189PC) FastUpload(ctx context.Context, dstDir model.Obj, file mode
if tmpF != nil {
writers = append(writers, tmpF)
}
pieceSHA1Hashes := make([]byte, 0, count*20)
written := int64(0)
for i := 1; i <= count; i++ {
if utils.IsCanceled(ctx) {
@@ -957,7 +966,17 @@ func (y *Cloud189PC) FastUpload(ctx context.Context, dstDir model.Obj, file mode
byteSize = lastSliceSize
}
n, err := utils.CopyWithBufferN(io.MultiWriter(writers...), file, byteSize)
// 如果需要生成 torrent,同时计算 SHA-1
var sha1Writer hash.Hash
var multiWriter io.Writer
if generateTorrent {
sha1Writer = sha1Pkg.New()
multiWriter = io.MultiWriter(append(writers, sha1Writer)...)
} else {
multiWriter = io.MultiWriter(writers...)
}
n, err := utils.CopyWithBufferN(multiWriter, file, byteSize)
written += n
if err != nil && err != io.EOF {
return nil, err
@@ -966,6 +985,11 @@ func (y *Cloud189PC) FastUpload(ctx context.Context, dstDir model.Obj, file mode
sliceMd5Hexs = append(sliceMd5Hexs, strings.ToUpper(hex.EncodeToString(md5Byte)))
partInfos = append(partInfos, fmt.Sprint(i, "-", base64.StdEncoding.EncodeToString(md5Byte)))
sliceMd5.Reset()
// 收集 SHA-1 piece hash(仅在本次分片实际写入了数据时追加)
if generateTorrent && n > 0 {
pieceSHA1Hashes = append(pieceSHA1Hashes, sha1Writer.Sum(nil)...)
}
}
if tmpF != nil {
@@ -1081,6 +1105,44 @@ func (y *Cloud189PC) FastUpload(ctx context.Context, dstDir model.Obj, file mode
if err != nil {
return nil, err
}
// 生成 torrent 文件(异步,不影响上传结果)
if generateTorrent && size > 0 && len(pieceSHA1Hashes) > 0 {
capturedDstDir := dstDir
capturedIsFamily := isFamily
capturedFileName := file.GetName()
go func() {
torrentData, err := GenerateTorrent(capturedFileName, size, fileMd5Hex, sliceMd5Hexs, sliceSize, pieceSHA1Hashes)
if err != nil {
utils.Log.Warnf("生成 torrent 失败: %v", err)
return
}
infoHash, _ := GetInfoHashHex(torrentData)
torrentName := capturedFileName + ".cas.torrent"
utils.Log.Infof("已生成 torrent: %s (info_hash: %s, size: %d bytes)",
torrentName, infoHash, len(torrentData))
// 将 torrent 文件上传到同一目录
torrentFileStream := &stream.FileStream{
Ctx: context.Background(),
Obj: &model.Object{
Name: torrentName,
Size: int64(len(torrentData)),
IsFolder: false,
},
Reader: bytes.NewReader(torrentData),
Mimetype: "application/x-bittorrent",
}
_, uploadErr := y.fastUpload(context.Background(), capturedDstDir, torrentFileStream, func(p float64) {}, capturedIsFamily, false, false)
if uploadErr != nil {
utils.Log.Warnf("上传 torrent 文件失败: %v", uploadErr)
} else {
utils.Log.Infof("torrent 文件已上传: %s", torrentName)
op.Cache.DeleteDirectory(y, capturedDstDir.GetPath())
}
}()
}
return resp.toFile(), nil
}
+2
View File
@@ -22,6 +22,7 @@ import (
_ "github.com/OpenListTeam/OpenList/v4/drivers/azure_blob"
_ "github.com/OpenListTeam/OpenList/v4/drivers/baidu_netdisk"
_ "github.com/OpenListTeam/OpenList/v4/drivers/baidu_photo"
_ "github.com/OpenListTeam/OpenList/v4/drivers/bunny_storage"
_ "github.com/OpenListTeam/OpenList/v4/drivers/chaoxing"
_ "github.com/OpenListTeam/OpenList/v4/drivers/chunk"
_ "github.com/OpenListTeam/OpenList/v4/drivers/cloudflare_imgbed"
@@ -41,6 +42,7 @@ import (
_ "github.com/OpenListTeam/OpenList/v4/drivers/github_releases"
_ "github.com/OpenListTeam/OpenList/v4/drivers/google_drive"
_ "github.com/OpenListTeam/OpenList/v4/drivers/google_photo"
_ "github.com/OpenListTeam/OpenList/v4/drivers/guangyapan"
_ "github.com/OpenListTeam/OpenList/v4/drivers/halalcloud"
_ "github.com/OpenListTeam/OpenList/v4/drivers/halalcloud_open"
_ "github.com/OpenListTeam/OpenList/v4/drivers/ilanzou"
+24
View File
@@ -2,6 +2,7 @@ package azure_blob
import (
"context"
"encoding/hex"
"fmt"
"io"
"path"
@@ -12,10 +13,13 @@ import (
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
log "github.com/sirupsen/logrus"
)
// Azure Blob Storage based on the blob APIs
@@ -283,6 +287,26 @@ func (d *AzureBlob) Put(ctx context.Context, dstDir model.Obj, stream model.File
// Determine optimal upload options based on file size
options := optimizedUploadOptions(stream.GetSize())
// Preserve the file content type for direct Azure Blob downloads.
contentType := stream.GetMimetype()
if contentType == "" {
contentType = "application/octet-stream"
}
options.HTTPHeaders = &blob.HTTPHeaders{
BlobContentType: &contentType,
}
// Preserve the MD5 checksum when it is available.
md5Hex := stream.GetHash().GetHash(utils.MD5)
if md5Hex != "" {
md5, err := hex.DecodeString(md5Hex)
if err == nil && len(md5) == 16 {
options.HTTPHeaders.BlobContentMD5 = md5
} else {
log.Warnf("Invalid MD5 hash: %s, error: %v", md5Hex, err)
}
}
// Track upload progress
progressTracker := &progressTracker{
total: stream.GetSize(),
+226
View File
@@ -0,0 +1,226 @@
package bunny_storage
import (
"bytes"
"context"
"fmt"
"net/http"
"net/url"
stdpath "path"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/go-resty/resty/v2"
)
type BunnyStorage struct {
model.Storage
Addition
client *resty.Client
endpoint *url.URL
cdnBase *url.URL
}
func (d *BunnyStorage) Config() driver.Config {
cfg := config
if d.StorageZoneName != "" && d.CDNBaseURL == "" {
cfg.OnlyProxy = true
cfg.PreferProxy = true
}
if d.CDNTokenKey != "" && d.CDNTokenIncludeIP {
cfg.LinkCacheMode = driver.LinkCacheIP
}
return cfg
}
func (d *BunnyStorage) GetAddition() driver.Additional {
return &d.Addition
}
func (d *BunnyStorage) Init(ctx context.Context) error {
if d.RootFolderPath == "" {
d.RootFolderPath = "/"
}
if d.Endpoint == "" {
d.Endpoint = defaultEndpoint
}
if d.SignURLExpire <= 0 {
d.SignURLExpire = 4
}
if d.CDNTokenMethod == "" {
d.CDNTokenMethod = cdnTokenMethodSHA256
}
endpoint, err := normalizeBaseURL(d.Endpoint, defaultEndpoint)
if err != nil {
return fmt.Errorf("invalid endpoint: %w", err)
}
d.endpoint = endpoint
if d.CDNBaseURL != "" {
cdnBase, err := normalizeBaseURL(d.CDNBaseURL, "")
if err != nil {
return fmt.Errorf("invalid cdn_base_url: %w", err)
}
d.cdnBase = cdnBase
}
d.client = base.RestyClient
if d.client == nil {
d.client = base.NewRestyClient()
}
return nil
}
func (d *BunnyStorage) Drop(ctx context.Context) error {
return nil
}
func (d *BunnyStorage) List(ctx context.Context, dir model.Obj, args model.ListArgs) ([]model.Obj, error) {
var items []bunnyObject
resp, err := d.authRequest().
SetContext(ctx).
SetResult(&items).
Get(d.storageURL(dir.GetPath(), true))
if err != nil {
return nil, err
}
if err := d.handleResponseError(resp); err != nil {
return nil, err
}
result := make([]model.Obj, 0, len(items))
placeholder := d.placeholderName()
for _, item := range items {
if item.ObjectName == "" {
continue
}
if !args.S3ShowPlaceholder && !item.IsDirectory && item.ObjectName == placeholder {
continue
}
result = append(result, d.toObj(dir.GetPath(), item))
}
return result, nil
}
func (d *BunnyStorage) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
if file.IsDir() {
return nil, errs.NotFile
}
cacheTTL := time.Duration(0)
if d.cdnBase != nil {
linkURL := d.cdnURL(d.cdnObjectPath(file.GetPath()))
link := &model.Link{
URL: linkURL,
ContentLength: file.GetSize(),
Expiration: &cacheTTL,
}
if d.CDNTokenKey != "" {
signedURL, _, err := d.signCDNURL(linkURL, args.IP)
if err != nil {
return nil, err
}
link.URL = signedURL
}
return link, nil
}
return &model.Link{
URL: d.storageURL(file.GetPath(), false),
Header: http.Header{"AccessKey": []string{d.AccessKey}},
ContentLength: file.GetSize(),
Expiration: &cacheTTL,
}, nil
}
func (d *BunnyStorage) MakeDir(ctx context.Context, parentDir model.Obj, dirName string) (model.Obj, error) {
dirPath := stdpath.Join(parentDir.GetPath(), dirName)
placeholderPath := stdpath.Join(dirPath, d.placeholderName())
if err := d.putReader(ctx, placeholderPath, bytes.NewReader(nil), 0, "application/octet-stream", nil); err != nil {
return nil, err
}
now := time.Now()
return &model.Object{
Path: dirPath,
Name: dirName,
Modified: now,
Ctime: now,
IsFolder: true,
}, nil
}
func (d *BunnyStorage) Remove(ctx context.Context, obj model.Obj) error {
resp, err := d.authRequest().
SetContext(ctx).
Delete(d.storageURL(obj.GetPath(), obj.IsDir()))
if err != nil {
return err
}
return d.handleResponseError(resp)
}
func (d *BunnyStorage) Put(ctx context.Context, dstDir model.Obj, file model.FileStreamer, up driver.UpdateProgress) (model.Obj, error) {
if up == nil {
up = func(float64) {}
}
dstPath := stdpath.Join(dstDir.GetPath(), file.GetName())
err := d.putReader(ctx, dstPath, driver.NewLimitedUploadStream(ctx, &driver.ReaderUpdatingProgress{
Reader: file,
UpdateProgress: up,
}), file.GetSize(), file.GetMimetype(), nil)
if err != nil {
return nil, err
}
now := time.Now()
return &model.Object{
Path: dstPath,
Name: file.GetName(),
Size: file.GetSize(),
Modified: now,
Ctime: now,
}, nil
}
func (d *BunnyStorage) putReader(ctx context.Context, path string, body any, size int64, contentType string, extraHeaders http.Header) error {
if contentType == "" {
contentType = "application/octet-stream"
}
req := d.authRequest().
SetContext(ctx).
SetBody(body).
SetHeader("Content-Type", contentType)
if size >= 0 {
req.SetHeader("Content-Length", fmt.Sprint(size))
}
for key, values := range extraHeaders {
for _, value := range values {
req.SetHeader(key, value)
}
}
resp, err := req.Put(d.storageURL(path, false))
if err != nil {
return err
}
return d.handleResponseError(resp)
}
func (d *BunnyStorage) Get(ctx context.Context, path string) (model.Obj, error) {
fullPath := stdpath.Join(d.GetRootPath(), path)
parentPath, name := stdpath.Split(fullPath)
parentPath = strings.TrimSuffix(parentPath, "/")
if parentPath == "" {
parentPath = "/"
}
objs, err := d.List(ctx, &model.Object{Path: parentPath, IsFolder: true}, model.ListArgs{S3ShowPlaceholder: true})
if err != nil {
return nil, err
}
for _, obj := range objs {
if obj.GetName() == name {
return obj, nil
}
}
return nil, errs.ObjectNotFound
}
var _ driver.Driver = (*BunnyStorage)(nil)
var _ driver.Getter = (*BunnyStorage)(nil)
+32
View File
@@ -0,0 +1,32 @@
package bunny_storage
import (
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/op"
)
type Addition struct {
driver.RootPath
StorageZoneName string `json:"storage_zone_name" required:"true"`
AccessKey string `json:"access_key" required:"true"`
Endpoint string `json:"endpoint" required:"true" default:"storage.bunnycdn.com"`
CDNBaseURL string `json:"cdn_base_url"`
CDNTokenKey string `json:"cdn_token_key"`
CDNTokenMethod string `json:"cdn_token_method" type:"select" options:"sha256,hmac_sha256" default:"sha256"`
CDNTokenIncludeIP bool `json:"cdn_token_include_ip" default:"false"`
SignURLExpire int `json:"sign_url_expire" type:"number" default:"4"`
Placeholder string `json:"placeholder" default:".openlist"`
}
var config = driver.Config{
Name: "Bunny Storage",
LocalSort: true,
DefaultRoot: "/",
CheckStatus: true,
}
func init() {
op.RegisterDriver(func() driver.Driver {
return &BunnyStorage{}
})
}
+27
View File
@@ -0,0 +1,27 @@
package bunny_storage
import "time"
type bunnyObject struct {
Guid string `json:"Guid"`
StorageZoneName string `json:"StorageZoneName"`
Path string `json:"Path"`
ObjectName string `json:"ObjectName"`
Length int64 `json:"Length"`
LastChanged string `json:"LastChanged"`
IsDirectory bool `json:"IsDirectory"`
ServerID int `json:"ServerId"`
UserID string `json:"UserId"`
DateCreated string `json:"DateCreated"`
StorageZoneID int64 `json:"StorageZoneId"`
}
type apiError struct {
HttpCode int `json:"HttpCode"`
Message string `json:"Message"`
}
type parsedTimes struct {
modified time.Time
created time.Time
}
+275
View File
@@ -0,0 +1,275 @@
package bunny_storage
import (
"crypto/hmac"
"crypto/sha256"
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"net/url"
stdpath "path"
"sort"
"strconv"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/go-resty/resty/v2"
)
const (
defaultEndpoint = "storage.bunnycdn.com"
defaultPlaceholder = ".openlist"
cdnTokenMethodSHA256 = "sha256"
cdnTokenMethodHMACSHA256 = "hmac_sha256"
)
func normalizeBaseURL(raw string, fallback string) (*url.URL, error) {
raw = strings.TrimSpace(raw)
if raw == "" {
raw = fallback
}
if raw == "" {
return nil, fmt.Errorf("empty url")
}
if !strings.Contains(raw, "://") {
raw = "https://" + raw
}
u, err := url.Parse(raw)
if err != nil {
return nil, err
}
if u.Host == "" {
return nil, fmt.Errorf("invalid url: %s", raw)
}
u.Path = strings.TrimRight(u.Path, "/")
return u, nil
}
func cleanObjectPath(path string) string {
if path == "" {
return "/"
}
return stdpath.Clean("/" + strings.TrimPrefix(path, "/"))
}
func stripObjectPathPrefix(path string, prefix string) (string, bool) {
path = cleanObjectPath(path)
prefix = cleanObjectPath(prefix)
if prefix == "/" {
return path, false
}
if path == prefix {
return "/", true
}
if strings.HasPrefix(path, prefix+"/") {
return cleanObjectPath(strings.TrimPrefix(path, prefix)), true
}
return path, false
}
func isObjectPathOrChild(path string, parent string) bool {
path = cleanObjectPath(path)
parent = cleanObjectPath(parent)
return path == parent || strings.HasPrefix(path, parent+"/")
}
func trimCDNBasePath(path string, mountPath string) string {
path = cleanObjectPath(path)
if path == "/" {
return ""
}
if stripped, ok := stripObjectPathPrefix(path, mountPath); ok {
path = stripped
}
if path == "/" {
return ""
}
return strings.TrimRight(path, "/")
}
func (d *BunnyStorage) cdnObjectPath(path string) string {
objectPath := cleanObjectPath(path)
if stripped, ok := stripObjectPathPrefix(objectPath, d.GetStorage().MountPath); ok {
objectPath = stripped
}
rootPath := cleanObjectPath(d.GetRootPath())
if rootPath != "/" && !isObjectPathOrChild(objectPath, rootPath) {
objectPath = cleanObjectPath(stdpath.Join(rootPath, objectPath))
}
return objectPath
}
func (d *BunnyStorage) placeholderName() string {
if d.Placeholder == "" {
return defaultPlaceholder
}
return d.Placeholder
}
func (d *BunnyStorage) storageURL(path string, dir bool) string {
u := *d.endpoint
cleanPath := cleanObjectPath(path)
zone := strings.Trim(d.StorageZoneName, "/")
if cleanPath == "/" {
u.Path = "/" + zone + "/"
return u.String()
}
u.Path = "/" + zone + "/" + strings.TrimPrefix(cleanPath, "/")
if dir && !strings.HasSuffix(u.Path, "/") {
u.Path += "/"
}
return u.String()
}
func (d *BunnyStorage) cdnURL(path string) string {
u := *d.cdnBase
cleanPath := cleanObjectPath(path)
basePath := trimCDNBasePath(u.Path, d.GetStorage().MountPath)
if cleanPath == "/" {
if basePath == "" {
u.Path = "/"
} else {
u.Path = basePath + "/"
}
return u.String()
}
u.Path = basePath + "/" + strings.TrimPrefix(cleanPath, "/")
return u.String()
}
func (d *BunnyStorage) authRequest() *resty.Request {
return d.client.R().SetHeader("AccessKey", d.AccessKey)
}
func (d *BunnyStorage) handleResponseError(resp *resty.Response) error {
if resp == nil {
return fmt.Errorf("empty response")
}
if resp.StatusCode() >= http.StatusOK && resp.StatusCode() < http.StatusMultipleChoices {
return nil
}
message := strings.TrimSpace(resp.String())
var apiErrors []apiError
if err := json.Unmarshal(resp.Body(), &apiErrors); err == nil && len(apiErrors) > 0 && apiErrors[0].Message != "" {
message = apiErrors[0].Message
}
switch resp.StatusCode() {
case http.StatusUnauthorized, http.StatusForbidden:
return errs.NewErr(errs.PermissionDenied, "bunny storage request failed: %s", message)
case http.StatusNotFound:
return errs.NewErr(errs.ObjectNotFound, "bunny storage request failed: %s", message)
default:
return fmt.Errorf("bunny storage request failed: %s: %s", resp.Status(), message)
}
}
func (d *BunnyStorage) parseTimes(item bunnyObject) parsedTimes {
return parsedTimes{
modified: parseBunnyTime(item.LastChanged, d.Modified),
created: parseBunnyTime(item.DateCreated, time.Time{}),
}
}
func parseBunnyTime(value string, fallback time.Time) time.Time {
if value == "" {
return fallback
}
if t, err := time.Parse(time.RFC3339Nano, value); err == nil {
return t
}
if t, err := time.Parse("2006-01-02T15:04:05.999999999", value); err == nil {
return t
}
if t, err := time.Parse("2006-01-02T15:04:05", value); err == nil {
return t
}
return fallback
}
func (d *BunnyStorage) toObj(parentPath string, item bunnyObject) model.Obj {
times := d.parseTimes(item)
return &model.Object{
ID: item.Guid,
Path: stdpath.Join(parentPath, item.ObjectName),
Name: item.ObjectName,
Size: item.Length,
Modified: times.modified,
Ctime: times.created,
IsFolder: item.IsDirectory,
}
}
func canonicalQuery(values url.Values) (string, error) {
keys := make([]string, 0, len(values))
for key := range values {
if key == "token" || key == "expires" {
continue
}
keys = append(keys, key)
}
sort.Strings(keys)
parts := make([]string, 0, len(keys))
for _, key := range keys {
vals := values[key]
if len(vals) > 1 {
return "", fmt.Errorf("duplicate query parameter %q is not supported", key)
}
value := ""
if len(vals) == 1 {
value = vals[0]
}
parts = append(parts, key+"="+value)
}
return strings.Join(parts, "&"), nil
}
func (d *BunnyStorage) signCDNURL(rawURL string, clientIP string) (string, time.Duration, error) {
return d.signCDNURLAt(rawURL, clientIP, time.Now())
}
func (d *BunnyStorage) signCDNURLAt(rawURL string, clientIP string, now time.Time) (string, time.Duration, error) {
expire := time.Hour * time.Duration(d.SignURLExpire)
if expire <= 0 {
expire = 4 * time.Hour
}
expires := now.Add(expire).Unix()
u, err := url.Parse(rawURL)
if err != nil {
return "", 0, err
}
query := u.Query()
parameterData, err := canonicalQuery(query)
if err != nil {
return "", 0, err
}
signaturePath, err := url.PathUnescape(u.EscapedPath())
if err != nil {
signaturePath = u.Path
}
if !d.CDNTokenIncludeIP {
clientIP = ""
}
token := d.signCDNToken(signaturePath, strconv.FormatInt(expires, 10), parameterData, clientIP)
query.Set("token", token)
query.Set("expires", strconv.FormatInt(expires, 10))
u.RawQuery = query.Encode()
return u.String(), expire, nil
}
func (d *BunnyStorage) signCDNToken(signaturePath string, expires string, parameterData string, clientIP string) string {
switch strings.ToLower(strings.TrimSpace(d.CDNTokenMethod)) {
case cdnTokenMethodHMACSHA256:
message := signaturePath + expires + parameterData + clientIP
mac := hmac.New(sha256.New, []byte(d.CDNTokenKey))
_, _ = mac.Write([]byte(message))
return "HS256-" + base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
default:
hashableBase := d.CDNTokenKey + signaturePath + expires + parameterData + clientIP
sum := sha256.Sum256([]byte(hashableBase))
return base64.RawURLEncoding.EncodeToString(sum[:])
}
}
+255
View File
@@ -0,0 +1,255 @@
package bunny_storage
import (
"context"
"net/url"
"testing"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/model"
)
func TestStorageURL(t *testing.T) {
endpoint, err := normalizeBaseURL("ny.storage.bunnycdn.com", defaultEndpoint)
if err != nil {
t.Fatal(err)
}
driver := &BunnyStorage{
Addition: Addition{
StorageZoneName: "my-zone",
},
endpoint: endpoint,
}
if got, want := driver.storageURL("/", true), "https://ny.storage.bunnycdn.com/my-zone/"; got != want {
t.Fatalf("root list url = %q, want %q", got, want)
}
if got, want := driver.storageURL("/dir/a file.txt", false), "https://ny.storage.bunnycdn.com/my-zone/dir/a%20file.txt"; got != want {
t.Fatalf("file url = %q, want %q", got, want)
}
if got, want := driver.storageURL("/dir", true), "https://ny.storage.bunnycdn.com/my-zone/dir/"; got != want {
t.Fatalf("dir url = %q, want %q", got, want)
}
}
func TestCDNURLWithBasePath(t *testing.T) {
cdnBase, err := normalizeBaseURL("https://cdn.example.com/prefix/", "")
if err != nil {
t.Fatal(err)
}
driver := &BunnyStorage{cdnBase: cdnBase}
if got, want := driver.cdnURL("/dir/a file.txt"), "https://cdn.example.com/prefix/dir/a%20file.txt"; got != want {
t.Fatalf("cdn url = %q, want %q", got, want)
}
}
func TestCDNURLUsesObjectPathWithoutMountPath(t *testing.T) {
cdnBase, err := normalizeBaseURL("https://cdn.firmant.me", "")
if err != nil {
t.Fatal(err)
}
driver := &BunnyStorage{
Storage: model.Storage{MountPath: "/BS"},
cdnBase: cdnBase,
}
if got, want := driver.cdnURL(driver.cdnObjectPath("/BS/Video")), "https://cdn.firmant.me/Video"; got != want {
t.Fatalf("cdn url = %q, want %q", got, want)
}
}
func TestCDNURLDropsMountPathFromBaseURL(t *testing.T) {
cdnBase, err := normalizeBaseURL("https://cdn.firmant.me/BS/", "")
if err != nil {
t.Fatal(err)
}
driver := &BunnyStorage{
Storage: model.Storage{MountPath: "/BS"},
cdnBase: cdnBase,
}
if got, want := driver.cdnURL(driver.cdnObjectPath("/Video")), "https://cdn.firmant.me/Video"; got != want {
t.Fatalf("cdn url = %q, want %q", got, want)
}
}
func TestCDNObjectPathKeepsRootFolderPath(t *testing.T) {
driver := &BunnyStorage{}
driver.MountPath = "/BS"
driver.RootFolderPath = "/library"
if got, want := driver.cdnObjectPath("/BS/Video"), "/library/Video"; got != want {
t.Fatalf("cdn object path = %q, want %q", got, want)
}
if got, want := driver.cdnObjectPath("/library/Video"), "/library/Video"; got != want {
t.Fatalf("cdn object path = %q, want %q", got, want)
}
}
func TestLinkDisablesLongLivedCache(t *testing.T) {
cdnBase, err := normalizeBaseURL("https://cdn.example.com", "")
if err != nil {
t.Fatal(err)
}
driver := &BunnyStorage{cdnBase: cdnBase}
link, err := driver.Link(context.Background(), &model.Object{
Path: "/video.mp4",
Name: "video.mp4",
Size: 123,
}, model.LinkArgs{})
if err != nil {
t.Fatal(err)
}
if link.Expiration == nil || *link.Expiration != 0 {
t.Fatalf("link expiration = %v, want immediate cache expiration", link.Expiration)
}
}
func TestSignCDNURL(t *testing.T) {
driver := &BunnyStorage{
Addition: Addition{
CDNTokenKey: "secret",
CDNTokenIncludeIP: true,
SignURLExpire: 1,
},
}
signed, expire, err := driver.signCDNURLAt("https://zone.b-cdn.net/video.mp4?quality=high", "192.0.2.1", time.Unix(1700000000, 0))
if err != nil {
t.Fatal(err)
}
if expire <= 0 {
t.Fatal("expected positive expiration")
}
parsed, err := url.Parse(signed)
if err != nil {
t.Fatal(err)
}
token := parsed.Query().Get("token")
if want := "FxSpFem88zFo6uHFziwTuoMQTgDaD2PEn5n1zTMBUBI"; token != want {
t.Fatalf("token = %q, want %q", token, want)
}
if parsed.Query().Get("expires") != "1700003600" {
t.Fatalf("expires = %q, want 1700003600", parsed.Query().Get("expires"))
}
if parsed.Query().Get("quality") != "high" {
t.Fatal("expected existing query parameters to be preserved")
}
}
func TestSignCDNURLSupportsHMACSHA256(t *testing.T) {
driver := &BunnyStorage{
Addition: Addition{
CDNTokenKey: "secret",
CDNTokenMethod: cdnTokenMethodHMACSHA256,
CDNTokenIncludeIP: true,
SignURLExpire: 1,
},
}
signed, _, err := driver.signCDNURLAt("https://zone.b-cdn.net/video.mp4?quality=high", "192.0.2.1", time.Unix(1700000000, 0))
if err != nil {
t.Fatal(err)
}
parsed, err := url.Parse(signed)
if err != nil {
t.Fatal(err)
}
if got, want := parsed.Query().Get("token"), "HS256-sdrSSJE2JVwhSk2AoDUrmTV1muH6R5UHpZVcVfHeNxg"; got != want {
t.Fatalf("token = %q, want %q", got, want)
}
}
func TestSignCDNURLUsesDecodedPathForSHA256(t *testing.T) {
driver := &BunnyStorage{
Addition: Addition{
CDNTokenKey: "secret",
SignURLExpire: 1,
},
}
signed, _, err := driver.signCDNURLAt("https://zone.b-cdn.net/%E8%A7%86%E9%A2%91/%5Ba%20b%5D.mp4", "", time.Unix(1700000000, 0))
if err != nil {
t.Fatal(err)
}
parsed, err := url.Parse(signed)
if err != nil {
t.Fatal(err)
}
if got, want := parsed.Query().Get("token"), "yq1evD7klw0e3DjCbv8dJptbW4S4JwVW3GKLnxfeKGM"; got != want {
t.Fatalf("token = %q, want %q", got, want)
}
}
func TestSignCDNURLTreatsPlusAsLiteralPathCharacter(t *testing.T) {
driver := &BunnyStorage{
Addition: Addition{
CDNTokenKey: "secret",
SignURLExpire: 1,
},
}
now := time.Unix(1700000000, 0)
literal, _, err := driver.signCDNURLAt("https://zone.b-cdn.net/a+b.mp4", "", now)
if err != nil {
t.Fatal(err)
}
encoded, _, err := driver.signCDNURLAt("https://zone.b-cdn.net/a%2Bb.mp4", "", now)
if err != nil {
t.Fatal(err)
}
space, _, err := driver.signCDNURLAt("https://zone.b-cdn.net/a%20b.mp4", "", now)
if err != nil {
t.Fatal(err)
}
literalURL, err := url.Parse(literal)
if err != nil {
t.Fatal(err)
}
encodedURL, err := url.Parse(encoded)
if err != nil {
t.Fatal(err)
}
spaceURL, err := url.Parse(space)
if err != nil {
t.Fatal(err)
}
if got, want := literalURL.Query().Get("token"), encodedURL.Query().Get("token"); got != want {
t.Fatalf("literal plus token = %q, encoded plus token = %q", got, want)
}
if got, notWant := literalURL.Query().Get("token"), spaceURL.Query().Get("token"); got == notWant {
t.Fatalf("literal plus token = %q, space token should differ", got)
}
}
func TestSignCDNURLRejectsDuplicateQueryParameters(t *testing.T) {
driver := &BunnyStorage{
Addition: Addition{
CDNTokenKey: "secret",
SignURLExpire: 1,
},
}
_, _, err := driver.signCDNURLAt("https://zone.b-cdn.net/video.mp4?quality=high&quality=low", "", time.Unix(1700000000, 0))
if err == nil {
t.Fatal("expected duplicate query parameters to be rejected")
}
if got, want := err.Error(), `duplicate query parameter "quality" is not supported`; got != want {
t.Fatalf("error = %q, want %q", got, want)
}
}
func TestParseBunnyTimeSupportsFractionalSecondsWithoutTimezone(t *testing.T) {
fallback := time.Unix(1, 0)
got := parseBunnyTime("2023-03-21T13:38:31.693", fallback)
want := time.Date(2023, 3, 21, 13, 38, 31, 693000000, time.UTC)
if !got.Equal(want) {
t.Fatalf("parsed time = %s, want %s", got.Format(time.RFC3339Nano), want.Format(time.RFC3339Nano))
}
}
func TestConfigProxyMode(t *testing.T) {
registeredConfig := (&BunnyStorage{}).Config()
if registeredConfig.OnlyProxy {
t.Fatal("driver registration config should allow users to choose proxy policy")
}
withoutCDN := (&BunnyStorage{Addition: Addition{StorageZoneName: "my-zone"}}).Config()
if !withoutCDN.OnlyProxy {
t.Fatal("storage API links require AccessKey headers and should be proxied without CDN")
}
withCDN := (&BunnyStorage{Addition: Addition{StorageZoneName: "my-zone", CDNBaseURL: "https://zone.b-cdn.net"}}).Config()
if withCDN.OnlyProxy {
t.Fatal("CDN links should be allowed to redirect directly")
}
}
+1 -1
View File
@@ -251,7 +251,7 @@ func (d *Crypt) Link(ctx context.Context, file model.Obj, _ model.LinkArgs) (*mo
rrf, err := stream.GetRangeReaderFromLink(remoteSize, remoteLink)
if err != nil {
_ = remoteLink.Close()
return nil, fmt.Errorf("the remote storage driver need to be enhanced to support encrytion")
return nil, fmt.Errorf("the remote storage driver need to be enhanced to support encryption")
}
mu := &sync.Mutex{}
+56 -64
View File
@@ -45,134 +45,126 @@ func (d *GithubReleases) List(ctx context.Context, dir model.Obj, args model.Lis
for i := range d.points {
point := &d.points[i]
if !d.Addition.ShowAllVersion { // latest
err := point.RequestRelease(d.GetRequest, args.Refresh)
if !d.Addition.ShowAllVersion {
// latest version mode
release, err := d.getLatestRelease(point.Repo)
if err != nil {
log.Warnf("failed to request release for %s: %v", point.Repo, err)
continue
}
if release == nil {
continue
}
if point.Point == path { // 与仓库路径相同
if point.Release == nil {
if err != nil {
return nil, fmt.Errorf("failed to get release for %s: %w", point.Repo, err)
}
return nil, fmt.Errorf("failed to get release for %s: unknown error", point.Repo)
}
files = append(files, point.GetLatestRelease()...)
if point.Point == path {
// 当前目录就是仓库挂载点
files = append(files, releaseToFiles(point.Point, release)...)
if d.Addition.ShowReadme {
otherFiles, err := point.GetOtherFile(d.GetRequest, args.Refresh)
if err != nil {
return nil, fmt.Errorf("failed to get other files for %s: %w", point.Repo, err)
other, err := d.fetchRepoFiles(point.Repo)
if err == nil {
files = append(files, otherFiles(point.Point, other)...)
} else {
log.Warnf("failed to get other files for %s: %v", point.Repo, err)
}
files = append(files, otherFiles...)
}
if d.Addition.ShowSourceCode {
files = append(files, point.GetSourceCode()...)
files = append(files, sourceCodeFiles(point.Point, release)...)
}
} else if strings.HasPrefix(point.Point, path) { // 仓库目录的父目录
} else if strings.HasPrefix(point.Point, path) {
// 仓库目录的父目录,需要聚合显示
nextDir := GetNextDir(point.Point, path)
if nextDir == "" {
continue
}
if err != nil {
return nil, fmt.Errorf("failed to get release for %s: %w", point.Repo, err)
}
hasSameDir := false
for index := range files {
if files[index].GetName() == nextDir {
hasSameDir = true
files[index].Size += point.GetLatestSize()
files[index].Size += releaseSize(release)
break
}
}
if !hasSameDir {
var updateAt, createAt string
if point.Release != nil {
updateAt = point.Release.PublishedAt
createAt = point.Release.CreatedAt
}
files = append(files, File{
Path: stdpath.Join(path, nextDir),
FileName: nextDir,
Size: point.GetLatestSize(),
UpdateAt: updateAt,
CreateAt: createAt,
Size: releaseSize(release),
UpdateAt: release.PublishedAt,
CreateAt: release.CreatedAt,
Type: "dir",
Url: "",
})
}
}
} else { // all version
err := point.RequestReleases(d.GetRequest, args.Refresh)
} else {
// all versions mode
releases, err := d.getAllReleases(point.Repo)
if err != nil {
log.Warnf("failed to request releases for %s: %v", point.Repo, err)
continue
}
if len(releases) == 0 {
// no releases but may still have repo files (e.g. README)
if point.Point == path && d.Addition.ShowReadme {
other, err := d.fetchRepoFiles(point.Repo)
if err == nil {
files = append(files, otherFiles(point.Point, other)...)
} else {
log.Warnf("failed to get other files for %s: %v", point.Repo, err)
}
}
continue
}
if point.Point == path { // 与仓库路径相同
if point.Releases == nil {
if err != nil {
return nil, fmt.Errorf("failed to get releases for %s: %w", point.Repo, err)
}
return nil, fmt.Errorf("failed to get releases for %s: unknown error", point.Repo)
}
files = append(files, point.GetAllVersion()...)
if point.Point == path {
// 当前目录就是仓库挂载点
files = append(files, releasesToVersionDirs(point.Point, releases)...)
if d.Addition.ShowReadme {
otherFiles, err := point.GetOtherFile(d.GetRequest, args.Refresh)
if err != nil {
return nil, fmt.Errorf("failed to get other files for %s: %w", point.Repo, err)
other, err := d.fetchRepoFiles(point.Repo)
if err == nil {
files = append(files, otherFiles(point.Point, other)...)
} else {
log.Warnf("failed to get other files for %s: %v", point.Repo, err)
}
files = append(files, otherFiles...)
}
} else if strings.HasPrefix(point.Point, path) { // 仓库目录的父目录
} else if strings.HasPrefix(point.Point, path) {
// 仓库目录的父目录
nextDir := GetNextDir(point.Point, path)
if nextDir == "" {
continue
}
if err != nil {
return nil, fmt.Errorf("failed to get releases for %s: %w", point.Repo, err)
}
hasSameDir := false
for index := range files {
if files[index].GetName() == nextDir {
hasSameDir = true
files[index].Size += point.GetAllVersionSize()
files[index].Size += releasesTotalSize(releases)
break
}
}
if !hasSameDir {
var updateAt, createAt string
if point.Releases != nil && len(*point.Releases) > 0 {
updateAt = (*point.Releases)[0].PublishedAt
createAt = (*point.Releases)[0].CreatedAt
}
files = append(files, File{
FileName: nextDir,
Path: stdpath.Join(path, nextDir),
Size: point.GetAllVersionSize(),
UpdateAt: updateAt,
CreateAt: createAt,
Size: releasesTotalSize(releases),
UpdateAt: releases[0].PublishedAt,
CreateAt: releases[0].CreatedAt,
Type: "dir",
Url: "",
})
}
} else if strings.HasPrefix(path, point.Point) { // 仓库目录的子目录
} else if strings.HasPrefix(path, point.Point) {
// 仓库目录的子目录(某个版本)
tagName := GetNextDir(path, point.Point)
if tagName == "" {
continue
}
if point.Releases == nil {
if err != nil {
return nil, fmt.Errorf("failed to get releases for %s: %w", point.Repo, err)
}
return nil, fmt.Errorf("failed to get releases for %s: unknown error", point.Repo)
}
files = append(files, point.GetReleaseByTagName(tagName)...)
files = append(files, releaseAssetsByTag(point.Point, tagName, releases)...)
if d.Addition.ShowSourceCode {
files = append(files, point.GetSourceCodeByTagName(tagName)...)
files = append(files, sourceCodeFilesByTag(point.Point, releases, tagName)...)
}
}
}
+3 -1
View File
@@ -12,7 +12,9 @@ type Addition struct {
Token string `json:"token" type:"string" required:"false" help:"GitHub token, if you want to access private repositories or increase the rate limit"`
ShowSourceCode bool `json:"show_source_code" type:"bool" default:"false" help:"show Source code (zip/tar.gz)"`
ShowAllVersion bool `json:"show_all_version" type:"bool" default:"false" help:"show all versions"`
GitHubProxy string `json:"gh_proxy" type:"string" default:"" help:"GitHub proxy, e.g. https://ghproxy.net/github.com or https://gh-proxy.com/github.com "`
PerPage int `json:"per_page" type:"number" default:"30" help:"releases per page (max 100), only works when show all versions"`
MaxPage int `json:"max_page" type:"number" default:"0" help:"max pages to fetch (0 = unlimited), only works when show all versions"`
GitHubProxy string `json:"gh_proxy" type:"string" default:"" help:"GitHub proxy, e.g. https://ghproxy.net/https://github.com or https://gh-proxy.com/https://github.com"`
}
var config = driver.Config{
+71 -173
View File
@@ -1,76 +1,28 @@
package github_releases
import (
"encoding/json"
"path"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/go-resty/resty/v2"
)
// MountPoint 表示一个仓库挂载点
type MountPoint struct {
Point string // 挂载点
Repo string // 仓库名 owner/repo
Release *Release // Release 指针 latest
Releases *[]Release // []Release 指针
OtherFile *[]FileInfo // 仓库根目录下的其他文件
Point string // 挂载点路径
Repo string // 仓库名 owner/repo
}
// 请求最新版本
func (m *MountPoint) RequestRelease(get func(url string) (*resty.Response, error), refresh bool) error {
if m.Repo == "" {
// Release 转为 File 列表
func releaseToFiles(point string, release *Release) []File {
if release == nil {
return nil
}
if m.Release == nil || refresh {
resp, err := get("https://api.github.com/repos/" + m.Repo + "/releases/latest")
if err != nil {
m.Release = nil
return err
}
release := new(Release)
if err := json.Unmarshal(resp.Body(), release); err != nil {
m.Release = nil
return err
}
m.Release = release
}
return nil
}
// 请求所有版本
func (m *MountPoint) RequestReleases(get func(url string) (*resty.Response, error), refresh bool) error {
if m.Repo == "" {
return nil
}
if m.Releases == nil || refresh {
resp, err := get("https://api.github.com/repos/" + m.Repo + "/releases")
if err != nil {
m.Releases = nil
return err
}
releases := new([]Release)
if err := json.Unmarshal(resp.Body(), releases); err != nil {
m.Releases = nil
return err
}
m.Releases = releases
}
return nil
}
// 获取最新版本
func (m *MountPoint) GetLatestRelease() []File {
if m.Release == nil {
return nil
}
files := make([]File, 0, len(m.Release.Assets))
for _, asset := range m.Release.Assets {
files := make([]File, 0, len(release.Assets))
for _, asset := range release.Assets {
files = append(files, File{
Path: path.Join(m.Point, asset.Name),
Path: path.Join(point, asset.Name),
FileName: asset.Name,
Size: asset.Size,
Type: "file",
@@ -82,53 +34,43 @@ func (m *MountPoint) GetLatestRelease() []File {
return files
}
// 获取最新版本大小
func (m *MountPoint) GetLatestSize() int64 {
if m.Release == nil {
// 计算 release 的 asset 总大小
func releaseSize(release *Release) int64 {
if release == nil {
return 0
}
size := int64(0)
for _, asset := range m.Release.Assets {
for _, asset := range release.Assets {
size += asset.Size
}
return size
}
// 获取所有版本
func (m *MountPoint) GetAllVersion() []File {
if m.Releases == nil {
return nil
}
files := make([]File, 0)
for _, release := range *m.Releases {
file := File{
Path: path.Join(m.Point, release.TagName),
// Releases 列表转为版本目录 File 列表
func releasesToVersionDirs(point string, releases []Release) []File {
files := make([]File, 0, len(releases))
for _, release := range releases {
files = append(files, File{
Path: path.Join(point, release.TagName),
FileName: release.TagName,
Size: m.GetSizeByTagName(release.TagName),
Size: releaseSize(&release),
Type: "dir",
UpdateAt: release.PublishedAt,
CreateAt: release.CreatedAt,
Url: release.HtmlUrl,
}
for _, asset := range release.Assets {
file.Size += asset.Size
}
files = append(files, file)
})
}
return files
}
// 根据版本号获取版本
func (m *MountPoint) GetReleaseByTagName(tagName string) []File {
if m.Releases == nil {
return nil
}
for _, item := range *m.Releases {
// 根据 tagName 查找 release 的 asset 文件列表
func releaseAssetsByTag(point, tagName string, releases []Release) []File {
for _, item := range releases {
if item.TagName == tagName {
files := make([]File, 0)
files := make([]File, 0, len(item.Assets))
for _, asset := range item.Assets {
files = append(files, File{
Path: path.Join(m.Point, tagName, asset.Name),
Path: path.Join(point, tagName, asset.Name),
FileName: asset.Name,
Size: asset.Size,
Type: "file",
@@ -143,118 +85,74 @@ func (m *MountPoint) GetReleaseByTagName(tagName string) []File {
return nil
}
// 根据版本号获取版本大小
func (m *MountPoint) GetSizeByTagName(tagName string) int64 {
if m.Releases == nil {
return 0
}
for _, item := range *m.Releases {
// 根据 tagName 计算 asset 总大小
func releasesSizeByTag(releases []Release, tagName string) int64 {
for _, item := range releases {
if item.TagName == tagName {
size := int64(0)
for _, asset := range item.Assets {
size += asset.Size
}
return size
return releaseSize(&item)
}
}
return 0
}
// 获取所有版本大小
func (m *MountPoint) GetAllVersionSize() int64 {
if m.Releases == nil {
return 0
}
// 计算所有 releases 的 asset 总大小
func releasesTotalSize(releases []Release) int64 {
size := int64(0)
for _, release := range *m.Releases {
for _, asset := range release.Assets {
size += asset.Size
}
for _, release := range releases {
size += releaseSize(&release)
}
return size
}
func (m *MountPoint) GetSourceCode() []File {
if m.Release == nil {
// Source code 文件
func sourceCodeFiles(point string, release *Release) []File {
if release == nil {
return nil
}
files := make([]File, 0)
// 无法获取文件大小,此处设为 1
files = append(files, File{
Path: path.Join(m.Point, "Source code (zip)"),
FileName: "Source code (zip)",
Size: 1,
Type: "file",
UpdateAt: m.Release.CreatedAt,
CreateAt: m.Release.CreatedAt,
Url: m.Release.ZipballUrl,
})
files = append(files, File{
Path: path.Join(m.Point, "Source code (tar.gz)"),
FileName: "Source code (tar.gz)",
Size: 1,
Type: "file",
UpdateAt: m.Release.CreatedAt,
CreateAt: m.Release.CreatedAt,
Url: m.Release.TarballUrl,
})
return files
return []File{
{
Path: path.Join(point, "Source code (zip)"),
FileName: "Source code (zip)",
Size: 1,
Type: "file",
UpdateAt: release.CreatedAt,
CreateAt: release.CreatedAt,
Url: release.ZipballUrl,
},
{
Path: path.Join(point, "Source code (tar.gz)"),
FileName: "Source code (tar.gz)",
Size: 1,
Type: "file",
UpdateAt: release.CreatedAt,
CreateAt: release.CreatedAt,
Url: release.TarballUrl,
},
}
}
func (m *MountPoint) GetSourceCodeByTagName(tagName string) []File {
if m.Releases == nil {
return nil
}
for _, item := range *m.Releases {
// 根据 tagName 获取 Source Code 文件
func sourceCodeFilesByTag(point string, releases []Release, tagName string) []File {
for _, item := range releases {
if item.TagName == tagName {
files := make([]File, 0)
files = append(files, File{
Path: path.Join(m.Point, "Source code (zip)"),
FileName: "Source code (zip)",
Size: 1,
Type: "file",
UpdateAt: item.CreatedAt,
CreateAt: item.CreatedAt,
Url: item.ZipballUrl,
})
files = append(files, File{
Path: path.Join(m.Point, "Source code (tar.gz)"),
FileName: "Source code (tar.gz)",
Size: 1,
Type: "file",
UpdateAt: item.CreatedAt,
CreateAt: item.CreatedAt,
Url: item.TarballUrl,
})
return files
return sourceCodeFiles(point, &item)
}
}
return nil
}
func (m *MountPoint) GetOtherFile(get func(url string) (*resty.Response, error), refresh bool) ([]File, error) {
if m.OtherFile == nil || refresh {
resp, err := get("https://api.github.com/repos/" + m.Repo + "/contents")
if err != nil {
m.OtherFile = nil
return nil, err
}
otherFile := new([]FileInfo)
if err := json.Unmarshal(resp.Body(), otherFile); err != nil {
m.OtherFile = nil
return nil, err
}
m.OtherFile = otherFile
}
// 仓库根目录下的 README/LICENSE 文件
func otherFiles(point string, fileInfos []FileInfo) []File {
files := make([]File, 0)
defaultTime := "1970-01-01T00:00:00Z"
for _, file := range *m.OtherFile {
if strings.HasSuffix(file.Name, ".md") || strings.HasPrefix(file.Name, "LICENSE") {
for _, file := range fileInfos {
if file.Type == "dir" {
continue
}
name := file.Name
if strings.EqualFold(name, "README.md") || strings.HasPrefix(name, "LICENSE") {
files = append(files, File{
Path: path.Join(m.Point, file.Name),
Path: path.Join(point, file.Name),
FileName: file.Name,
Size: file.Size,
Type: "file",
@@ -264,7 +162,7 @@ func (m *MountPoint) GetOtherFile(get func(url string) (*resty.Response, error),
})
}
}
return files, nil
return files
}
type File struct {
+78 -10
View File
@@ -1,8 +1,8 @@
package github_releases
import (
"encoding/json"
"fmt"
"path/filepath"
"strings"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
@@ -49,10 +49,8 @@ func (d *GithubReleases) ParseRepos(text string) ([]MountPoint, error) {
}
points = append(points, MountPoint{
Point: path,
Repo: repo,
Release: nil,
Releases: nil,
Point: path,
Repo: repo,
})
}
d.points = points
@@ -76,9 +74,79 @@ func GetNextDir(wholePath string, basePath string) string {
return ""
}
// 判断当前目录是否是目标目录的祖先目录
func IsAncestorDir(parentDir string, targetDir string) bool {
absTargetDir, _ := filepath.Abs(targetDir)
absParentDir, _ := filepath.Abs(parentDir)
return strings.HasPrefix(absTargetDir, absParentDir)
// getLatestRelease 获取最新 release
func (d *GithubReleases) getLatestRelease(repo string) (*Release, error) {
resp, err := d.GetRequest("https://api.github.com/repos/" + repo + "/releases/latest")
if err != nil {
return nil, err
}
release := new(Release)
if err := json.Unmarshal(resp.Body(), release); err != nil {
return nil, err
}
return release, nil
}
// getAllReleases 获取所有 releases(支持自动翻页)
func (d *GithubReleases) getAllReleases(repo string) ([]Release, error) {
perPage := d.Addition.PerPage
if perPage < 1 {
perPage = 30
} else if perPage > 100 {
perPage = 100
}
maxPage := d.Addition.MaxPage
if maxPage < 0 {
maxPage = 0
}
allReleases := make([]Release, 0)
page := 1
for {
url := fmt.Sprintf("https://api.github.com/repos/%s/releases?per_page=%d&page=%d", repo, perPage, page)
resp, err := d.GetRequest(url)
if err != nil {
return nil, err
}
releases := make([]Release, 0)
if err := json.Unmarshal(resp.Body(), &releases); err != nil {
return nil, err
}
if len(releases) == 0 {
break
}
allReleases = append(allReleases, releases...)
// 达到最大页数限制
if maxPage > 0 && page >= maxPage {
break
}
// 如果返回数量小于 perPage,说明是最后一页
if len(releases) < perPage {
break
}
page++
}
return allReleases, nil
}
// fetchRepoFiles 获取仓库根目录文件列表
func (d *GithubReleases) fetchRepoFiles(repo string) ([]FileInfo, error) {
resp, err := d.GetRequest("https://api.github.com/repos/" + repo + "/contents")
if err != nil {
return nil, err
}
files := make([]FileInfo, 0)
if err := json.Unmarshal(resp.Body(), &files); err != nil {
return nil, err
}
return files, nil
}
+786
View File
@@ -0,0 +1,786 @@
package guangyapan
import (
"context"
"errors"
"fmt"
"strings"
"sync"
"time"
"github.com/OpenListTeam/OpenList/v4/drivers/base"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
"github.com/go-resty/resty/v2"
log "github.com/sirupsen/logrus"
)
const (
accountBaseURL = "https://account.guangyapan.com"
apiBaseURL = "https://api.guangyapan.com"
)
type GuangYaPan struct {
model.Storage
Addition
accountClient *resty.Client
apiClient *resty.Client
resolvedRootFolderID string
rootFolderResolved bool
// refreshMu protects concurrent access to AccessToken/RefreshToken during refresh.
refreshMu sync.Mutex
// statusTimer tracks delayed status updates for cancellation on Drop.
statusTimer *time.Timer
// apiRateLimit throttles requests per API endpoint so that batch operations
// (e.g. copying many files cross-storage) don't flood the upstream API.
apiRateLimit sync.Map
}
// apiRateInterval is the minimum gap between two requests to the same endpoint.
const apiRateInterval = 500 * time.Millisecond
func (d *GuangYaPan) Config() driver.Config {
return config
}
func (d *GuangYaPan) GetAddition() driver.Additional {
return &d.Addition
}
func (d *GuangYaPan) Init(ctx context.Context) error {
d.ClientID = strings.TrimSpace(d.ClientID)
if d.ClientID == "" {
return errors.New("client_id is required, please provide a valid client_id")
}
d.DeviceID = normalizeDeviceID(d.DeviceID)
if d.DeviceID == "" {
d.DeviceID = randomDeviceID()
}
deviceSign := strings.TrimSpace(d.DeviceSign)
if deviceSign == "" {
deviceSign = "wdi10." + d.DeviceID
}
if d.PageSize <= 0 {
d.PageSize = 100
}
if d.OrderBy < 0 {
d.OrderBy = 3
}
if d.SortType != 0 && d.SortType != 1 {
d.SortType = 1
}
d.RootPath = strings.TrimSpace(d.RootPath)
d.AccessToken = strings.TrimSpace(d.AccessToken)
d.RefreshToken = strings.TrimSpace(d.RefreshToken)
d.PhoneNumber = strings.TrimSpace(d.PhoneNumber)
d.VerifyCode = strings.TrimSpace(d.VerifyCode)
d.CaptchaToken = strings.TrimSpace(d.CaptchaToken)
d.VerificationID = strings.TrimSpace(d.VerificationID)
d.resolvedRootFolderID = ""
d.rootFolderResolved = false
d.accountClient = base.NewRestyClient().
SetBaseURL(accountBaseURL).
SetHeader("Accept", "application/json, text/plain, */*").
SetHeader("Content-Type", "application/json").
SetHeader("X-Device-Model", "chrome%2F147.0.0.0").
SetHeader("X-Device-Name", "PC-Chrome").
SetHeader("X-Device-Sign", deviceSign).
SetHeader("X-Net-Work-Type", "NONE").
SetHeader("X-OS-Version", "MacIntel").
SetHeader("X-Platform-Version", "1").
SetHeader("X-Protocol-Version", "301").
SetHeader("X-Provider-Name", "NONE").
SetHeader("X-SDK-Version", "9.0.2").
SetHeader("X-Client-Id", d.ClientID).
SetHeader("X-Client-Version", "0.0.1").
SetHeader("X-Device-Id", d.DeviceID)
d.apiClient = base.NewRestyClient().
SetBaseURL(apiBaseURL).
SetHeader("Accept", "application/json, text/plain, */*").
SetHeader("Content-Type", "application/json").
SetHeader("Did", d.DeviceID).
SetHeader("Dt", "4")
// Priority: access_token -> refresh_token -> sms login.
if d.AccessToken != "" {
if err := d.validateToken(ctx); err == nil {
return d.prepareRootFolder(ctx)
}
d.AccessToken = ""
}
if d.RefreshToken != "" {
if err := d.refreshToken(ctx); err == nil {
if err2 := d.validateToken(ctx); err2 == nil {
return d.prepareRootFolder(ctx)
}
}
}
// Two-stage SMS flow:
// 1) phone only + send_code=true: send code and cache verification_id (do not fail init).
// 2) phone + verify_code: complete login and save tokens.
if d.PhoneNumber != "" {
if d.canSMSLogin() {
if err := d.loginBySMSCode(ctx); err != nil {
return err
}
if err := d.validateToken(ctx); err != nil {
return err
}
return d.prepareRootFolder(ctx)
}
if d.SendCode {
d.setTempStatus("SMS sending in progress...")
if err := d.prepareSMSCode(ctx); err != nil {
d.setTempStatus(fmt.Sprintf("SMS send failed: %v. Please check captcha/meta and set send_code=true to retry.", err))
log.Warnf("guangyapan: prepare sms code failed: %v", err)
} else {
d.setTempStatus("SMS sent successfully. Please fill verify_code and save to complete login.")
}
}
return nil
}
return errors.New("login failed: provide a valid access_token, or refresh_token, or phone_number + verify_code + captcha_token")
}
func (d *GuangYaPan) Drop(ctx context.Context) error {
if d.statusTimer != nil {
d.statusTimer.Stop()
}
return nil
}
func (d *GuangYaPan) GetRoot(ctx context.Context) (model.Obj, error) {
rootID, err := d.getRootFolderID(ctx)
if err != nil {
return nil, err
}
return &model.Object{
ID: rootID,
Path: "/",
Name: "root",
Size: 0,
Modified: d.Modified,
IsFolder: true,
}, nil
}
func (d *GuangYaPan) List(ctx context.Context, dir model.Obj, args model.ListArgs) ([]model.Obj, error) {
if err := d.ensureAccessToken(ctx); err != nil {
return nil, err
}
parentID := dir.GetID()
const maxPage = 10000
res := make([]model.Obj, 0, d.PageSize)
for page := 0; page < maxPage; page++ {
var resp listResp
body := map[string]any{
"parentId": parentID,
"page": page,
"pageSize": d.PageSize,
"orderBy": d.OrderBy,
"sortType": d.SortType,
}
if err := d.postAPI(ctx, "/userres/v1/file/get_file_list", body, &resp); err != nil {
return nil, err
}
for _, item := range resp.Data.List {
res = append(res, &model.Object{
ID: item.FileID,
Path: parentID,
Name: item.FileName,
Size: item.FileSize,
Modified: unixOrZero(item.UTime),
Ctime: unixOrZero(item.CTime),
IsFolder: item.ResType == 2,
})
}
if len(resp.Data.List) < d.PageSize {
break
}
if resp.Data.Total > 0 && len(res) >= resp.Data.Total {
break
}
}
return res, nil
}
func (d *GuangYaPan) Link(ctx context.Context, file model.Obj, args model.LinkArgs) (*model.Link, error) {
if file.IsDir() {
return nil, errs.NotFile
}
if err := d.ensureAccessToken(ctx); err != nil {
return nil, err
}
var resp downloadResp
if err := d.postAPI(ctx, "/nd.bizuserres.s/v1/get_res_download_url", map[string]any{
"fileId": file.GetID(),
}, &resp); err != nil {
return nil, err
}
url := strings.TrimSpace(resp.Data.SignedURL)
if url == "" {
url = strings.TrimSpace(resp.Data.DownloadURL)
}
if url == "" {
return nil, errors.New("empty download url")
}
return &model.Link{URL: url}, nil
}
func (d *GuangYaPan) MakeDir(ctx context.Context, parentDir model.Obj, dirName string) error {
if err := d.ensureAccessToken(ctx); err != nil {
return err
}
name := strings.TrimSpace(dirName)
if name == "" {
return errors.New("dir name is empty")
}
parentID := parentDir.GetID()
var out createDirResp
if err := d.postAPI(ctx, "/nd.bizuserres.s/v1/file/create_dir", map[string]any{
"parentId": parentID,
"dirName": name,
}, &out); err != nil {
return err
}
if !isSuccessMsg(out.Msg) {
return fmt.Errorf("make dir failed: %s", strings.TrimSpace(out.Msg))
}
return nil
}
func (d *GuangYaPan) Rename(ctx context.Context, srcObj model.Obj, newName string) error {
if err := d.ensureAccessToken(ctx); err != nil {
return err
}
fileID := strings.TrimSpace(srcObj.GetID())
if fileID == "" {
return errors.New("file id is empty")
}
name := strings.TrimSpace(newName)
if name == "" {
return errors.New("new name is empty")
}
var out commonResp
if err := d.postAPI(ctx, "/nd.bizuserres.s/v1/file/rename", map[string]any{
"fileId": fileID,
"newName": name,
}, &out); err != nil {
return err
}
if !isSuccessMsg(out.Msg) {
return fmt.Errorf("rename failed: %s", strings.TrimSpace(out.Msg))
}
return nil
}
func (d *GuangYaPan) Remove(ctx context.Context, obj model.Obj) error {
if err := d.ensureAccessToken(ctx); err != nil {
return err
}
fileID := strings.TrimSpace(obj.GetID())
if fileID == "" {
return errors.New("file id is empty")
}
var del taskResp
if err := d.postAPI(ctx, "/nd.bizuserres.s/v1/file/delete_file", map[string]any{
"fileIds": []string{fileID},
}, &del); err != nil {
return err
}
if !isSuccessMsg(del.Msg) {
return fmt.Errorf("delete failed: %s", strings.TrimSpace(del.Msg))
}
taskID := strings.TrimSpace(del.Data.TaskID)
if taskID == "" {
// Some backends may apply deletion synchronously.
return nil
}
return d.waitTaskDone(ctx, taskID)
}
func (d *GuangYaPan) Move(ctx context.Context, srcObj, dstDir model.Obj) error {
if err := d.ensureAccessToken(ctx); err != nil {
return err
}
fileID := strings.TrimSpace(srcObj.GetID())
if fileID == "" {
return errors.New("file id is empty")
}
parentID := dstDir.GetID()
var out taskResp
if err := d.postAPI(ctx, "/nd.bizuserres.s/v1/file/move_file", map[string]any{
"fileIds": []string{fileID},
"parentId": parentID,
}, &out); err != nil {
return err
}
if !isSuccessMsg(out.Msg) {
return fmt.Errorf("move failed: %s", strings.TrimSpace(out.Msg))
}
taskID := strings.TrimSpace(out.Data.TaskID)
if taskID == "" {
return nil
}
return d.waitTaskDone(ctx, taskID)
}
func (d *GuangYaPan) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
if err := d.ensureAccessToken(ctx); err != nil {
return err
}
fileID := strings.TrimSpace(srcObj.GetID())
if fileID == "" {
return errors.New("file id is empty")
}
parentID := dstDir.GetID()
var out taskResp
if err := d.postAPI(ctx, "/nd.bizuserres.s/v1/file/copy_file", map[string]any{
"fileIds": []string{fileID},
"parentId": parentID,
}, &out); err != nil {
return err
}
if !isSuccessMsg(out.Msg) {
return fmt.Errorf("copy failed: %s", strings.TrimSpace(out.Msg))
}
taskID := strings.TrimSpace(out.Data.TaskID)
if taskID == "" {
return nil
}
return d.waitTaskDone(ctx, taskID)
}
func (d *GuangYaPan) Put(ctx context.Context, dstDir model.Obj, file model.FileStreamer, up driver.UpdateProgress) (model.Obj, error) {
if err := d.ensureAccessToken(ctx); err != nil {
return nil, err
}
if file == nil {
return nil, errors.New("file is nil")
}
if file.GetSize() < 0 {
return nil, errors.New("invalid file size")
}
name := strings.TrimSpace(file.GetName())
if name == "" {
return nil, errors.New("file name is empty")
}
parentID := dstDir.GetID()
token, code, err := d.getUploadToken(ctx, parentID, name, file.GetSize())
if err != nil {
return nil, err
}
taskID := strings.TrimSpace(token.TaskID)
// code == 156 (instant upload) or AlreadyDone mean the backend has already
// finished/imported the file; there is no OSS upload to perform.
if code == 156 || token.AlreadyDone {
if taskID == "" {
return nil, errors.New("instant upload returns empty task id")
}
if err := d.waitUploadTaskInfo(ctx, taskID); err != nil {
return nil, err
}
return nil, nil
}
if token.ObjectPath == "" || token.BucketName == "" || token.EndPoint == "" || token.AccessKeyID == "" || token.SecretAccessKey == "" {
return nil, errors.New("upload token is incomplete")
}
ossEndpoint := normalizeOSSEndpoint(token.EndPoint, token.BucketName)
client, err := oss.New(ossEndpoint, token.AccessKeyID, token.SecretAccessKey, oss.SecurityToken(token.SessionToken))
if err != nil {
return nil, fmt.Errorf("create oss client failed: %w", err)
}
bucket, err := client.Bucket(token.BucketName)
if err != nil {
return nil, fmt.Errorf("create oss bucket failed: %w", err)
}
if file.GetSize() == 0 {
if err := bucket.PutObject(token.ObjectPath, strings.NewReader("")); err != nil {
return nil, err
}
} else {
if err := d.multipartUploadToOSS(ctx, bucket, token.ObjectPath, file, up); err != nil {
return nil, err
}
}
if taskID == "" {
return nil, nil
}
if err := d.waitUploadTaskInfo(ctx, taskID); err != nil {
return nil, err
}
return nil, nil
}
func (d *GuangYaPan) GetDetails(ctx context.Context) (*model.StorageDetails, error) {
if err := d.ensureAccessToken(ctx); err != nil {
return nil, err
}
var resp assetsInfoResp
if err := d.postAPI(ctx, "/nd.bizassets.s/v1/get_assets", nil, &resp); err != nil {
return nil, err
}
if resp.IsSuccess() && resp.Data.TotalSpaceSize > 0 {
return &model.StorageDetails{
DiskUsage: model.DiskUsage{
TotalSpace: resp.Data.TotalSpaceSize,
UsedSpace: resp.Data.UsedSpaceSize,
},
}, nil
}
return nil, errors.New("failed to get storage details")
}
func (d *GuangYaPan) getRootFolderID(ctx context.Context) (string, error) {
if d.rootFolderResolved {
return d.resolvedRootFolderID, nil
}
if err := d.ensureAccessToken(ctx); err != nil {
return "", err
}
if err := d.prepareRootFolder(ctx); err != nil {
return "", err
}
return d.resolvedRootFolderID, nil
}
func (d *GuangYaPan) prepareRootFolder(ctx context.Context) error {
rootID, err := d.resolveConfiguredRootFolderID(ctx)
if err != nil {
return err
}
d.resolvedRootFolderID = rootID
d.rootFolderResolved = true
return nil
}
func (d *GuangYaPan) resolveConfiguredRootFolderID(ctx context.Context) (string, error) {
root := strings.TrimSpace(d.RootPath)
if root == "" {
return "", nil
}
return d.resolveFolderPath(ctx, root)
}
func (d *GuangYaPan) resolveFolderPath(ctx context.Context, rootPath string) (string, error) {
cleanPath := strings.Trim(strings.ReplaceAll(strings.TrimSpace(rootPath), "\\", "/"), "/")
if cleanPath == "" {
return "", nil
}
parentID := ""
for _, name := range strings.Split(cleanPath, "/") {
if name == "" {
continue
}
childID, err := d.findChildFolderID(ctx, parentID, name)
if err != nil {
return "", err
}
parentID = childID
}
return parentID, nil
}
func (d *GuangYaPan) findChildFolderID(ctx context.Context, parentID, name string) (string, error) {
pageSize := d.PageSize
if pageSize <= 0 {
pageSize = 100
}
const maxPage = 10000
seen := 0
for page := 0; page < maxPage; page++ {
var resp listResp
body := map[string]any{
"parentId": parentID,
"page": page,
"pageSize": pageSize,
"orderBy": d.OrderBy,
"sortType": d.SortType,
}
if err := d.postAPI(ctx, "/nd.bizuserres.s/v1/file/get_file_list", body, &resp); err != nil {
return "", err
}
for _, item := range resp.Data.List {
seen++
if item.ResType == 2 && item.FileName == name {
return item.FileID, nil
}
}
if len(resp.Data.List) < pageSize {
break
}
if resp.Data.Total > 0 && seen >= resp.Data.Total {
break
}
}
if parentID == "" {
return "", fmt.Errorf("resolve root folder path failed: folder %q not found under /", name)
}
return "", fmt.Errorf("resolve root folder path failed: folder %q not found under parent %s", name, parentID)
}
func (d *GuangYaPan) ensureAccessToken(ctx context.Context) error {
if strings.TrimSpace(d.AccessToken) != "" {
return nil
}
if strings.TrimSpace(d.RefreshToken) == "" {
return errors.New("not logged in, please re-init storage")
}
return d.refreshToken(ctx)
}
func (d *GuangYaPan) validateToken(ctx context.Context) error {
var me userMeResp
resp, err := d.accountClient.R().
SetContext(ctx).
SetHeader("Authorization", "Bearer "+d.AccessToken).
SetResult(&me).
Get("/v1/user/me")
if err != nil {
return err
}
if resp.IsError() {
return fmt.Errorf("validate token failed: status=%d body=%s", resp.StatusCode(), resp.String())
}
if strings.TrimSpace(me.Sub) == "" {
return errors.New("validate token failed: empty user sub")
}
return nil
}
func (d *GuangYaPan) refreshToken(ctx context.Context) error {
if strings.TrimSpace(d.RefreshToken) == "" {
return errors.New("refresh_token is empty")
}
d.refreshMu.Lock()
defer d.refreshMu.Unlock()
// Double-check after acquiring lock (may have been refreshed by another goroutine)
if strings.TrimSpace(d.AccessToken) != "" {
if err := d.validateToken(ctx); err == nil {
return nil
}
}
var out tokenResp
resp, err := d.accountClient.R().
SetContext(ctx).
SetBody(map[string]any{
"client_id": d.ClientID,
"grant_type": "refresh_token",
"refresh_token": d.RefreshToken,
}).
SetResult(&out).
Post("/v1/auth/token")
if err != nil {
return err
}
if resp.IsError() || out.Error != "" || strings.TrimSpace(out.AccessToken) == "" {
errMsg := strings.TrimSpace(out.ErrorDesc)
if errMsg == "" {
errMsg = strings.TrimSpace(out.Error)
}
if errMsg == "" {
errMsg = strings.TrimSpace(resp.String())
}
if errMsg == "" {
errMsg = fmt.Sprintf("status=%d", resp.StatusCode())
}
return fmt.Errorf("refresh token failed: %s", errMsg)
}
d.AccessToken = strings.TrimSpace(out.AccessToken)
if strings.TrimSpace(out.RefreshToken) != "" {
d.RefreshToken = strings.TrimSpace(out.RefreshToken)
}
op.MustSaveDriverStorage(d)
return nil
}
func (d *GuangYaPan) canSMSLogin() bool {
return d.PhoneNumber != "" && d.VerifyCode != ""
}
func (d *GuangYaPan) loginBySMSCode(ctx context.Context) error {
verificationID := strings.TrimSpace(d.VerificationID)
if verificationID == "" {
var err error
verificationID, err = d.requestVerificationID(ctx)
if err != nil {
return err
}
}
var step2 verifyResp
resp, err := d.accountClient.R().
SetContext(ctx).
SetBody(map[string]any{
"verification_id": verificationID,
"verification_code": d.VerifyCode,
"client_id": d.ClientID,
}).
SetResult(&step2).
Post("/v1/auth/verification/verify")
if err != nil {
return err
}
if resp.IsError() || step2.Error != "" || strings.TrimSpace(step2.VerificationToken) == "" {
return fmt.Errorf("verify code failed: %s", d.accountErr(step2.ErrorDesc, step2.Error, resp))
}
var out tokenResp
resp, err = d.accountClient.R().
SetContext(ctx).
SetBody(map[string]any{
"verification_code": d.VerifyCode,
"verification_token": step2.VerificationToken,
"username": normalizePhoneE164(d.PhoneNumber),
"client_id": d.ClientID,
}).
SetResult(&out).
Post("/v1/auth/signin")
if err != nil {
return err
}
if resp.IsError() || out.Error != "" || strings.TrimSpace(out.AccessToken) == "" {
return fmt.Errorf("signin failed: %s", d.accountErr(out.ErrorDesc, out.Error, resp))
}
d.AccessToken = strings.TrimSpace(out.AccessToken)
d.RefreshToken = strings.TrimSpace(out.RefreshToken)
d.VerificationID = ""
// One-time SMS code should not be reused after successful login.
d.VerifyCode = ""
op.MustSaveDriverStorage(d)
return nil
}
func (d *GuangYaPan) prepareSMSCode(ctx context.Context) error {
// Explicit send action should always refresh verification_id.
d.VerificationID = ""
if err := d.ensureCaptchaToken(ctx, false); err != nil {
return err
}
verificationID, err := d.requestVerificationID(ctx)
if err != nil {
return err
}
d.VerificationID = verificationID
d.SendCode = false
op.MustSaveDriverStorage(d)
return nil
}
func (d *GuangYaPan) requestVerificationID(ctx context.Context) (string, error) {
req := d.accountClient.R().SetContext(ctx)
if d.CaptchaToken != "" {
req.SetHeader("X-Captcha-Token", d.CaptchaToken)
}
var step1 verificationResp
resp, err := req.
SetBody(map[string]any{
"phone_number": normalizePhoneE164(d.PhoneNumber),
"target": "ANY",
"client_id": d.ClientID,
}).
SetResult(&step1).
Post("/v1/auth/verification")
if err != nil {
return "", err
}
if resp.IsError() || step1.Error != "" || strings.TrimSpace(step1.VerificationID) == "" {
// If captcha token is expired/invalid, refresh it once and retry.
if strings.Contains(step1.Error, "captcha_invalid") || strings.Contains(step1.ErrorDesc, "captcha_token expired") {
if err := d.ensureCaptchaToken(ctx, true); err == nil {
return d.requestVerificationID(ctx)
}
}
return "", fmt.Errorf("request verification failed: %s", d.accountErr(step1.ErrorDesc, step1.Error, resp))
}
return strings.TrimSpace(step1.VerificationID), nil
}
func (d *GuangYaPan) ensureCaptchaToken(ctx context.Context, force bool) error {
if !force && d.CaptchaToken != "" {
return nil
}
var out captchaInitResp
req := d.accountClient.R().SetContext(ctx)
if d.CaptchaToken != "" {
req.SetHeader("X-Captcha-Token", d.CaptchaToken)
}
resp, err := req.
SetBody(map[string]any{
"client_id": d.ClientID,
"action": "POST:/v1/auth/verification",
"device_id": d.DeviceID,
"meta": map[string]any{
"username": normalizePhoneE164(d.PhoneNumber),
"phone_number": normalizePhoneE164(d.PhoneNumber),
"VERIFICATION_PHONE": normalizePhoneE164(d.PhoneNumber),
},
}).
SetResult(&out).
Post("/v1/shield/captcha/init")
if err != nil {
return err
}
if resp.IsError() || out.Error != "" || strings.TrimSpace(out.CaptchaToken) == "" {
return fmt.Errorf("init captcha token failed: %s", d.accountErr(out.ErrorDesc, out.Error, resp))
}
d.CaptchaToken = strings.TrimSpace(out.CaptchaToken)
op.MustSaveDriverStorage(d)
return nil
}
// Interface compliance checks
var (
_ driver.Driver = (*GuangYaPan)(nil)
_ driver.GetRooter = (*GuangYaPan)(nil)
_ driver.Mkdir = (*GuangYaPan)(nil)
_ driver.Move = (*GuangYaPan)(nil)
_ driver.Copy = (*GuangYaPan)(nil)
_ driver.Rename = (*GuangYaPan)(nil)
_ driver.Remove = (*GuangYaPan)(nil)
_ driver.PutResult = (*GuangYaPan)(nil)
_ driver.WithDetails = (*GuangYaPan)(nil)
)
+37
View File
@@ -0,0 +1,37 @@
package guangyapan
import (
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/op"
)
type Addition struct {
RootPath string `json:"root_path" help:"Full path in GuangYaPan cloud drive"`
PhoneNumber string `json:"phone_number" type:"text" help:"Phone number for SMS login, e.g. +86 13800000000"`
CaptchaToken string `json:"captcha_token" help:"Captcha token required by /v1/auth/verification"`
SendCode bool `json:"send_code" type:"bool" help:"Set true and save to send SMS code, it auto-resets to false after sending"`
VerifyCode string `json:"verify_code" type:"text" help:"SMS verification code used with phone_number; fill then save to finish login"`
VerificationID string `json:"verification_id" type:"text" help:"Auto-generated after sending SMS code; do not edit manually"`
AccessToken string `json:"access_token" help:"Bearer access token (optional if refresh_token is provided)"`
RefreshToken string `json:"refresh_token" help:"Refresh token for auto-login/auto-refresh"`
ClientID string `json:"client_id" required:"true" help:"Client ID for GuangYaPan API, must be provided"`
DeviceID string `json:"device_id" help:"Optional custom device id (32 hex chars), auto-generated when empty"`
DeviceSign string `json:"device_sign" help:"Optional custom X-Device-Sign header (generated from device_id when empty)"`
PageSize int `json:"page_size" type:"number" default:"100"`
OrderBy int `json:"order_by" type:"number" options:"0,1,2,3,4" default:"3" help:"Sort field used by the file list"`
SortType int `json:"sort_type" type:"number" options:"0,1" default:"1" help:"Sort direction used by the file list"`
}
var config = driver.Config{
Name: "GuangYaPan",
DefaultRoot: "",
CheckStatus: true,
Alert: "info|Two-stage SMS login: (1) fill phone_number (+ captcha_token if needed), set send_code=true and save; (2) fill verify_code and save to finish login and auto-save access_token/refresh_token.",
NoOverwriteUpload: true,
}
func init() {
op.RegisterDriver(func() driver.Driver {
return &GuangYaPan{}
})
}
+169
View File
@@ -0,0 +1,169 @@
package guangyapan
import (
"context"
"errors"
"fmt"
"net/url"
stdpath "path"
"strings"
"github.com/OpenListTeam/OpenList/v4/internal/model"
)
func (d *GuangYaPan) ResolveOfflineResource(ctx context.Context, fileURL string) (*OfflineResolveData, error) {
if err := d.ensureAccessToken(ctx); err != nil {
return nil, err
}
fileURL = strings.TrimSpace(fileURL)
if fileURL == "" {
return nil, errors.New("offline url is empty")
}
var resp offlineResolveResp
if err := d.postAPI(ctx, "/cloudcollection/v1/resolve_res", map[string]any{
"url": fileURL,
}, &resp); err != nil {
return nil, err
}
if !isSuccessMsg(resp.Msg) {
return nil, fmt.Errorf("resolve offline resource failed: %s", strings.TrimSpace(resp.Msg))
}
return &resp.Data, nil
}
func (d *GuangYaPan) OfflineDownload(ctx context.Context, fileURL string, parentDir model.Obj, fileName string) (*OfflineTask, error) {
resolved, err := d.ResolveOfflineResource(ctx, fileURL)
if err != nil {
return nil, err
}
parentID := parentDir.GetID()
rootID, err := d.getRootFolderID(ctx)
if err != nil {
return nil, err
}
if parentID == rootID {
parentID = ""
}
taskURL := strings.TrimSpace(resolved.URL)
if taskURL == "" {
taskURL = strings.TrimSpace(fileURL)
}
name := strings.TrimSpace(fileName)
if name == "" {
name = resolved.defaultName(taskURL)
}
body := map[string]any{
"url": taskURL,
"parentId": parentID,
"newName": name,
}
if indexes := resolved.fileIndexes(); len(indexes) > 0 {
body["fileIndexes"] = indexes
}
var resp offlineCreateResp
if err := d.postAPI(ctx, "/cloudcollection/v1/create_task", body, &resp); err != nil {
return nil, err
}
if !isSuccessMsg(resp.Msg) {
return nil, fmt.Errorf("create offline task failed: %s", strings.TrimSpace(resp.Msg))
}
taskID := strings.TrimSpace(resp.Data.TaskID)
if taskID == "" {
return nil, errors.New("create offline task failed: empty task id")
}
return &OfflineTask{
TaskID: taskID,
FileName: name,
Res: taskURL,
}, nil
}
func (d *GuangYaPan) OfflineList(ctx context.Context, taskIDs []string, statuses []int, cursor string, pageSize int) ([]OfflineTask, error) {
if err := d.ensureAccessToken(ctx); err != nil {
return nil, err
}
body := map[string]any{}
if len(taskIDs) > 0 {
body["taskIds"] = taskIDs
}
if len(statuses) > 0 {
body["status"] = statuses
}
if cursor = strings.TrimSpace(cursor); cursor != "" {
body["cursor"] = cursor
}
if pageSize > 0 {
body["pageSize"] = pageSize
}
var resp offlineListResp
if err := d.postAPI(ctx, "/cloudcollection/v1/list_task", body, &resp); err != nil {
return nil, err
}
if !isSuccessMsg(resp.Msg) {
return nil, fmt.Errorf("list offline tasks failed: %s", strings.TrimSpace(resp.Msg))
}
return resp.Data.List, nil
}
func (d *GuangYaPan) DeleteOfflineTasks(ctx context.Context, taskIDs []string) error {
if err := d.ensureAccessToken(ctx); err != nil {
return err
}
if len(taskIDs) == 0 {
return nil
}
var resp offlineDeleteResp
if err := d.postAPI(ctx, "/cloudcollection/v2/delete_task", map[string]any{
"taskIds": taskIDs,
}, &resp); err != nil {
return err
}
if !isSuccessMsg(resp.Msg) {
return fmt.Errorf("delete offline tasks failed: %s", strings.TrimSpace(resp.Msg))
}
return nil
}
func (d OfflineResolveData) defaultName(fileURL string) string {
if d.BTResInfo != nil && strings.TrimSpace(d.BTResInfo.FileName) != "" {
return strings.TrimSpace(d.BTResInfo.FileName)
}
u, err := url.Parse(fileURL)
if err == nil {
name := strings.TrimSpace(stdpath.Base(u.Path))
if name != "" && name != "." && name != "/" {
if decoded, err := url.PathUnescape(name); err == nil {
name = decoded
}
return name
}
}
return "offline_download"
}
func (d OfflineResolveData) fileIndexes() []int {
if d.BTResInfo == nil || len(d.BTResInfo.Subfiles) == 0 {
return nil
}
indexes := make([]int, 0, len(d.BTResInfo.Subfiles))
for i, file := range d.BTResInfo.Subfiles {
if file.FileIndex != nil {
indexes = append(indexes, *file.FileIndex)
continue
}
indexes = append(indexes, i)
}
return indexes
}
func isSuccessMsg(msg string) bool {
msg = strings.TrimSpace(msg)
return msg == "" || strings.EqualFold(msg, "success")
}
+232
View File
@@ -0,0 +1,232 @@
package guangyapan
import "time"
type tokenResp struct {
AccessToken string `json:"access_token"`
RefreshToken string `json:"refresh_token"`
TokenType string `json:"token_type"`
ExpiresIn int64 `json:"expires_in"`
Sub string `json:"sub"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
type verificationResp struct {
VerificationID string `json:"verification_id"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
type captchaInitResp struct {
CaptchaToken string `json:"captcha_token"`
ExpiresIn int64 `json:"expires_in"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
type verifyResp struct {
VerificationToken string `json:"verification_token"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
type userMeResp struct {
Sub string `json:"sub"`
}
type listResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
Total int `json:"total"`
List []fileItem `json:"list"`
} `json:"data"`
}
type fileItem struct {
FileID string `json:"fileId"`
ParentID string `json:"parentId"`
FileName string `json:"fileName"`
FileSize int64 `json:"fileSize"`
ResType int `json:"resType"`
CTime int64 `json:"ctime"`
UTime int64 `json:"utime"`
}
type downloadResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
SignedURL string `json:"signedURL"`
DownloadURL string `json:"downloadUrl"`
} `json:"data"`
}
type createDirResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
FileID string `json:"fileId"`
FileName string `json:"fileName"`
ResType int `json:"resType"`
CTime int64 `json:"ctime"`
UTime int64 `json:"utime"`
} `json:"data"`
}
type commonResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
}
// taskResp is used for async operations (delete, move, copy) that return a task ID.
type taskResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
TaskID string `json:"taskId"`
} `json:"data"`
}
type taskStatusResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
Status int `json:"status"`
} `json:"data"`
}
type uploadTokenResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data uploadTokenData `json:"data"`
}
type uploadTokenData struct {
TaskID string `json:"taskId"`
AlreadyDone bool `json:"-"`
ObjectPath string `json:"objectPath"`
Provider any `json:"provider"`
Region string `json:"region"`
BucketName string `json:"bucketName"`
EndPoint string `json:"endPoint"`
FullEndPoint string `json:"fullEndPoint"`
CallbackVar string `json:"callbackVar"`
AccessKeyID string `json:"accessKeyID"`
SecretAccessKey string `json:"secretAccessKey"`
SessionToken string `json:"sessionToken"`
Creds struct {
AccessKeyID string `json:"accessKeyID"`
SecretAccessKey string `json:"secretAccessKey"`
SessionToken string `json:"sessionToken"`
} `json:"creds"`
}
type taskInfoResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
FileID string `json:"fileId"`
} `json:"data"`
}
// assetsInfoResp is the response for storage details query.
type assetsInfoResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
TotalSpaceSize int64 `json:"totalSpaceSize"`
UsedSpaceSize int64 `json:"usedSpaceSize"`
} `json:"data"`
}
func (r assetsInfoResp) IsSuccess() bool {
return isSuccessMsg(r.Msg)
}
// Offline download types
type offlineResolveResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data OfflineResolveData `json:"data"`
}
type OfflineResolveData struct {
ResType int `json:"resType"`
BTResInfo *OfflineBTResInfo `json:"btResInfo"`
URL string `json:"url"`
}
type OfflineBTResInfo struct {
InfoHash string `json:"infoHash"`
FileName string `json:"fileName"`
FileSize int64 `json:"fileSize"`
SubfilesNum int `json:"subfilesNum"`
Subfiles []OfflineSubfile `json:"subfiles"`
CreateTime int64 `json:"createTime"`
ExcludeIndices []int `json:"excludeIndices"`
}
type OfflineSubfile struct {
FileName string `json:"fileName"`
FileIndex *int `json:"fileIndex"`
FileSize int64 `json:"fileSize"`
}
type offlineCreateResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
TaskID string `json:"taskId"`
URL string `json:"url"`
} `json:"data"`
}
type offlineDeleteResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
TaskIDs []string `json:"taskIds"`
} `json:"data"`
}
type offlineListResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
StatusCounts []struct {
Status int `json:"status"`
Count int `json:"count"`
} `json:"statusCounts"`
Cursor string `json:"cursor"`
List []OfflineTask `json:"list"`
Total int `json:"total"`
} `json:"data"`
}
type OfflineTask struct {
TaskID string `json:"taskId"`
FileName string `json:"fileName"`
TotalSize int64 `json:"totalSize"`
Status int `json:"status"`
CreateTime int64 `json:"createTime"`
Res string `json:"res"`
ResType int `json:"resType"`
Progress int `json:"progress"`
FileID string `json:"fileId"`
IsDir bool `json:"isDir"`
Exist bool `json:"exist"`
}
func unixOrZero(v int64) time.Time {
if v <= 0 {
return time.Time{}
}
return time.Unix(v, 0)
}
+407
View File
@@ -0,0 +1,407 @@
package guangyapan
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"io"
"net/url"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
streamPkg "github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
"github.com/avast/retry-go"
"github.com/go-resty/resty/v2"
"golang.org/x/time/rate"
)
// --- HTTP request helpers ---
func (d *GuangYaPan) accountErr(desc, short string, resp *resty.Response) string {
msg := strings.TrimSpace(desc)
if msg == "" {
msg = strings.TrimSpace(short)
}
if msg == "" && resp != nil {
msg = strings.TrimSpace(resp.String())
}
if msg == "" && resp != nil {
msg = fmt.Sprintf("status=%d", resp.StatusCode())
}
if msg == "" {
msg = "unknown error"
}
return msg
}
func (d *GuangYaPan) apiRateLimitWait(ctx context.Context, path string) error {
value, _ := d.apiRateLimit.LoadOrStore(path, rate.NewLimiter(rate.Every(apiRateInterval), 1))
return value.(*rate.Limiter).Wait(ctx)
}
func (d *GuangYaPan) postAPI(ctx context.Context, path string, body any, out any) error {
if err := d.ensureAccessToken(ctx); err != nil {
return err
}
if err := d.apiRateLimitWait(ctx, path); err != nil {
return err
}
resp, err := d.apiClient.R().
SetContext(ctx).
SetHeader("Authorization", "Bearer "+d.AccessToken).
SetBody(body).
SetResult(out).
Post(path)
if err != nil {
return err
}
if resp.StatusCode() == 401 || resp.StatusCode() == 403 {
if strings.TrimSpace(d.RefreshToken) == "" {
return fmt.Errorf("request failed: status=%d body=%s", resp.StatusCode(), resp.String())
}
if err := d.refreshToken(ctx); err != nil {
return err
}
resp, err = d.apiClient.R().
SetContext(ctx).
SetHeader("Authorization", "Bearer "+d.AccessToken).
SetBody(body).
SetResult(out).
Post(path)
if err != nil {
return err
}
}
if resp.IsError() {
return fmt.Errorf("request failed: status=%d body=%s", resp.StatusCode(), resp.String())
}
return nil
}
func (d *GuangYaPan) setTempStatus(status string) {
if d.statusTimer != nil {
d.statusTimer.Stop()
}
// initStorage sets status to WORK after Init returns, so we update it shortly after.
d.statusTimer = time.AfterFunc(200*time.Millisecond, func() {
d.GetStorage().SetStatus(status)
op.MustSaveDriverStorage(d)
})
}
// --- Task polling helpers ---
func (d *GuangYaPan) waitTaskDone(ctx context.Context, taskID string) error {
const (
maxTry = 30
interval = 300 * time.Millisecond
)
for i := 0; i < maxTry; i++ {
var out taskStatusResp
if err := d.postAPI(ctx, "/nd.bizuserres.s/v1/get_task_status", map[string]any{
"taskId": taskID,
}, &out); err != nil {
return err
}
if !isSuccessMsg(out.Msg) {
return fmt.Errorf("get task status failed: %s", strings.TrimSpace(out.Msg))
}
switch out.Data.Status {
case 2:
return nil
case -1, 3:
return fmt.Errorf("task %s failed with status=%d", taskID, out.Data.Status)
}
if i == maxTry-1 {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(interval):
}
}
return fmt.Errorf("task %s timeout", taskID)
}
// --- Upload helpers ---
func (d *GuangYaPan) getUploadToken(ctx context.Context, parentID, name string, size int64) (*uploadTokenData, int, error) {
var out uploadTokenResp
err := d.postAPI(ctx, "/nd.bizuserres.s/v1/get_res_center_token", map[string]any{
"capacity": 2,
"name": name,
"parentId": parentID,
"res": map[string]any{
"fileSize": size,
},
}, &out)
if err != nil {
return nil, 0, err
}
msg := strings.TrimSpace(out.Msg)
if !isSuccessMsg(msg) && !isUploadAlreadyDone(msg) {
return nil, out.Code, fmt.Errorf("get upload token failed: %s", msg)
}
if out.Data.TaskID == "" {
return nil, out.Code, errors.New("get upload token failed: empty task id")
}
// When the backend reports the file is already uploaded/instant-uploaded,
// it returns a valid TaskID without OSS credentials.
// Mark it so the caller can skip the real upload and just wait for the task.
if out.Code == 156 || isUploadAlreadyDone(msg) {
out.Data.AlreadyDone = true
}
if out.Data.AccessKeyID == "" {
out.Data.AccessKeyID = out.Data.Creds.AccessKeyID
}
if out.Data.SecretAccessKey == "" {
out.Data.SecretAccessKey = out.Data.Creds.SecretAccessKey
}
if out.Data.SessionToken == "" {
out.Data.SessionToken = out.Data.Creds.SessionToken
}
if strings.TrimSpace(out.Data.EndPoint) == "" {
out.Data.EndPoint = strings.TrimSpace(out.Data.FullEndPoint)
}
if strings.TrimSpace(out.Data.EndPoint) != "" && !strings.HasPrefix(out.Data.EndPoint, "http://") && !strings.HasPrefix(out.Data.EndPoint, "https://") {
if strings.TrimSpace(out.Data.FullEndPoint) != "" {
out.Data.EndPoint = strings.TrimSpace(out.Data.FullEndPoint)
} else if strings.TrimSpace(out.Data.BucketName) != "" {
host := strings.TrimSpace(out.Data.EndPoint)
prefix := strings.TrimSpace(out.Data.BucketName) + "."
if strings.HasPrefix(host, prefix) {
out.Data.EndPoint = "https://" + host
} else {
out.Data.EndPoint = "https://" + strings.TrimSpace(out.Data.BucketName) + "." + host
}
} else {
out.Data.EndPoint = "https://" + strings.TrimSpace(out.Data.EndPoint)
}
}
return &out.Data, out.Code, nil
}
func (d *GuangYaPan) waitUploadTaskInfo(ctx context.Context, taskID string) error {
const (
maxTry = 300
interval = 1 * time.Second
)
for i := 0; i < maxTry; i++ {
var out taskInfoResp
if err := d.postAPI(ctx, "/nd.bizuserres.s/v1/file/get_info_by_task_id", map[string]any{
"taskId": taskID,
}, &out); err != nil {
return err
}
if out.Data.FileID != "" {
return nil
}
switch out.Code {
case 145, 146, 147, 155, 163, 0:
// uploading/verifying/processing
default:
if strings.TrimSpace(out.Msg) != "" {
return fmt.Errorf("upload task failed: code=%d msg=%s", out.Code, strings.TrimSpace(out.Msg))
}
}
if i == maxTry-1 {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(interval):
}
}
return fmt.Errorf("upload task %s timeout", taskID)
}
func (d *GuangYaPan) multipartUploadToOSS(ctx context.Context, bucket *oss.Bucket, objectPath string, file model.FileStreamer, up driver.UpdateProgress) error {
partSize := calcUploadPartSize(file.GetSize())
imur, err := bucket.InitiateMultipartUpload(objectPath, oss.Sequential())
if err != nil {
return err
}
total := file.GetSize()
partCount := int((total + partSize - 1) / partSize)
// Use StreamSectionReader for seekable, retryable chunk reads (hybrid cache).
ss, err := streamPkg.NewStreamSectionReader(file, int(partSize), &up)
if err != nil {
return err
}
parts := make([]oss.UploadPart, 0, partCount)
for i := 0; i < partCount; i++ {
if utils.IsCanceled(ctx) {
return ctx.Err()
}
offset := int64(i) * partSize
length := partSize
if remain := total - offset; length > remain {
length = remain
}
rd, err := ss.GetSectionReader(offset, length)
if err != nil {
return err
}
var part oss.UploadPart
err = retry.Do(func() error {
rd.Seek(0, io.SeekStart)
var uploadErr error
part, uploadErr = bucket.UploadPart(imur, driver.NewLimitedUploadStream(ctx, rd), length, i+1)
return uploadErr
},
retry.Context(ctx),
retry.Attempts(3),
retry.DelayType(retry.BackOffDelay),
retry.Delay(time.Second))
ss.FreeSectionReader(rd)
if err != nil {
return fmt.Errorf("failed to upload part %d: %w", i+1, err)
}
parts = append(parts, part)
}
_, err = bucket.CompleteMultipartUpload(imur, parts)
return err
}
// --- Normalization helpers ---
func normalizeOSSEndpoint(endpoint, bucket string) string {
ep := strings.TrimSpace(endpoint)
if ep == "" {
return ep
}
if !strings.HasPrefix(ep, "http://") && !strings.HasPrefix(ep, "https://") {
ep = "https://" + ep
}
u, err := url.Parse(ep)
if err != nil || u.Host == "" {
return ep
}
host := u.Host
prefix := strings.TrimSpace(bucket)
if prefix != "" && strings.HasPrefix(host, prefix+".") {
host = strings.TrimPrefix(host, prefix+".")
}
u.Host = host
return u.String()
}
func normalizeDeviceID(v string) string {
v = strings.ToLower(strings.TrimSpace(v))
v = strings.ReplaceAll(v, "-", "")
if len(v) != 32 {
return ""
}
for _, ch := range v {
if (ch < '0' || ch > '9') && (ch < 'a' || ch > 'f') {
return ""
}
}
return v
}
func randomDeviceID() string {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
return "0123456789abcdef0123456789abcdef"
}
return hex.EncodeToString(b)
}
func normalizeCaptchaUsername(phone string) string {
p := strings.TrimSpace(phone)
p = strings.ReplaceAll(p, " ", "")
p = strings.TrimPrefix(p, "+")
// Keep only digits.
b := make([]rune, 0, len(p))
for _, ch := range p {
if ch >= '0' && ch <= '9' {
b = append(b, ch)
}
}
digits := string(b)
// Mainland number normalization: +86xxxxxxxxxxx -> xxxxxxxxxxx
if strings.HasPrefix(digits, "86") && len(digits) > 11 {
digits = digits[2:]
}
return digits
}
func normalizePhoneE164(phone string) string {
p := strings.TrimSpace(phone)
if p == "" {
return ""
}
p = strings.ReplaceAll(p, " ", "")
if strings.HasPrefix(p, "+") {
// Format as "+86 1xxxxxxxxxx" to match browser payload expectations.
if strings.HasPrefix(p, "+86") && len(p) > 3 {
rest := strings.TrimPrefix(p, "+86")
return "+86 " + rest
}
return p
}
// If raw mainland number is provided, normalize with +86 prefix.
digits := normalizeCaptchaUsername(p)
if len(digits) == 11 {
return "+86 " + digits
}
return p
}
func calcUploadPartSize(size int64) int64 {
const (
mb = int64(1024 * 1024)
gb = int64(1024 * 1024 * 1024)
)
switch {
case size <= 100*mb:
return 1 * mb
case size <= 16*gb:
return 2 * mb
case size <= 160*gb:
return 4 * mb
default:
return 8 * mb
}
}
// isUploadAlreadyDone reports whether the upload-token response indicates the
// file was already uploaded (instant upload). In that case the backend returns
// a valid TaskID but no OSS credentials, and we should just wait for the task
// instead of starting a real upload.
func isUploadAlreadyDone(msg string) bool {
msg = strings.TrimSpace(msg)
if msg == "" {
return false
}
if strings.EqualFold(msg, "上传已完成") {
return true
}
if strings.EqualFold(msg, "upload completed") {
return true
}
if strings.EqualFold(msg, "already uploaded") {
return true
}
if strings.EqualFold(msg, "秒传成功") {
return true
}
return false
}
+1 -1
View File
@@ -304,7 +304,7 @@ var findFromReg = regexp.MustCompile(`data : '(.+?)'`) // 查找from字符串
func htmlFormToMap(html string) (map[string]string, error) {
forms := findFromReg.FindStringSubmatch(html)
if len(forms) != 2 {
return nil, fmt.Errorf("not find file sgin")
return nil, fmt.Errorf("not find file sign")
}
return formToMap(forms[1]), nil
}
+2 -2
View File
@@ -132,7 +132,7 @@ type FileOrFolderByShareUrl struct {
//T int `json:"t"`
// 文件夹特有
IsFloder bool `json:"-"`
IsFolder bool `json:"-"`
//
Url string `json:"-"`
@@ -162,7 +162,7 @@ func (f *FileOrFolderByShareUrl) GetSize() int64 {
}
return *f.size
}
func (f *FileOrFolderByShareUrl) IsDir() bool { return f.IsFloder }
func (f *FileOrFolderByShareUrl) IsDir() bool { return f.IsFolder }
func (f *FileOrFolderByShareUrl) ModTime() time.Time {
if f.time == nil {
time := MustParseTime(f.Time)
+42 -24
View File
@@ -157,25 +157,43 @@ func (d *LanZou) request(url string, method string, callback base.ReqCallback, u
}
func (d *LanZou) Login() ([]*http.Cookie, error) {
resp, err := base.NewRestyClient().SetRedirectPolicy(resty.NoRedirectPolicy()).
R().SetFormData(map[string]string{
"task": "3",
"uid": d.Account,
"pwd": d.Password,
"setSessionId": "",
"setSig": "",
"setScene": "",
"setTocen": "",
"formhash": "",
}).Post("https://up.woozooo.com/mlogin.php")
if err != nil {
return nil, err
var vs string
for retry := 0; retry < 3; retry++ {
req := base.NewRestyClient().SetRedirectPolicy(resty.NoRedirectPolicy()).R()
// 如果已计算出 acw_sc__v2,通过 cookie 携带
if vs != "" {
req.SetHeader("cookie", "acw_sc__v2="+vs)
}
resp, err := req.SetFormData(map[string]string{
"task": "3",
"uid": d.Account,
"pwd": d.Password,
"setSessionId": "",
"setSig": "",
"setScene": "",
"setTocen": "",
"formhash": "",
}).Post("https://up.woozooo.com/mlogin.php")
if err != nil {
return nil, err
}
bodyStr := resp.String()
if strings.Contains(bodyStr, "acw_sc__v2") {
vs, err = CalcAcwScV2(bodyStr)
if err != nil {
return nil, err
}
continue
}
if utils.Json.Get(resp.Body(), "zt").ToInt() != 1 {
return nil, fmt.Errorf("login err: %s", resp.Body())
}
d.Cookie = CookieToString(resp.Cookies())
return resp.Cookies(), nil
}
if utils.Json.Get(resp.Body(), "zt").ToInt() != 1 {
return nil, fmt.Errorf("login err: %s", resp.Body())
}
d.Cookie = CookieToString(resp.Cookies())
return resp.Cookies(), nil
return nil, errors.New("acw_sc__v2 validation error")
}
/*
@@ -572,14 +590,14 @@ func (d *LanZou) getFolderByShareUrl(pwd string, sharePageData string) ([]FileOr
files := make([]FileOrFolderByShareUrl, 0)
// vip获取文件夹
floders := findSubFolderReg.FindAllStringSubmatch(sharePageData, -1)
for _, floder := range floders {
if len(floder) == 3 {
folders := findSubFolderReg.FindAllStringSubmatch(sharePageData, -1)
for _, folder := range folders {
if len(folder) == 3 {
files = append(files, FileOrFolderByShareUrl{
// Pwd: pwd, // 子文件夹不加密
ID: floder[1],
NameAll: floder[2],
IsFloder: true,
ID: folder[1],
NameAll: folder[2],
IsFolder: true,
})
}
}
+12 -4
View File
@@ -178,7 +178,7 @@ func (d *S3) Move(ctx context.Context, srcObj, dstDir model.Obj) error {
}
func (d *S3) Rename(ctx context.Context, srcObj model.Obj, newName string) error {
err := d.copy(ctx, srcObj.GetPath(), stdpath.Join(stdpath.Dir(srcObj.GetPath()), newName), srcObj.IsDir())
err := d.copy(ctx, srcObj.GetPath(), stdpath.Join(stdpath.Dir(srcObj.GetPath()), newName), srcObj.GetSize(), srcObj.IsDir())
if err != nil {
return err
}
@@ -186,14 +186,14 @@ func (d *S3) Rename(ctx context.Context, srcObj model.Obj, newName string) error
}
func (d *S3) Copy(ctx context.Context, srcObj, dstDir model.Obj) error {
return d.copy(ctx, srcObj.GetPath(), stdpath.Join(dstDir.GetPath(), srcObj.GetName()), srcObj.IsDir())
return d.copy(ctx, srcObj.GetPath(), stdpath.Join(dstDir.GetPath(), srcObj.GetName()), srcObj.GetSize(), srcObj.IsDir())
}
func (d *S3) Remove(ctx context.Context, obj model.Obj) error {
if obj.IsDir() {
return d.removeDir(ctx, obj.GetPath())
}
return d.removeFile(obj.GetPath())
return d.removeFile(ctx, obj.GetPath())
}
func (d *S3) Put(ctx context.Context, dstDir model.Obj, s model.FileStreamer, up driver.UpdateProgress) error {
@@ -249,7 +249,15 @@ func (d *S3) GetDirectUploadInfo(ctx context.Context, _ string, dstDir model.Obj
// implements driver.Getter interface
func (d *S3) Get(ctx context.Context, path string) (model.Obj, error) {
// try to get object as a file using HeadObject
path = stdpath.Join(d.GetRootPath(), path)
rootPath := d.GetRootPath()
// Avoid double-prepending root path when path already contains it.
// This happens when obj.GetPath() from a previous Get call is passed back
// to op.List → op.Get → d.Get.
if rootPath != "/" && utils.IsSubPath(rootPath, path) {
// path already includes the root prefix
} else {
path = stdpath.Join(rootPath, path)
}
key := getKey(path, false)
headInput := &s3.HeadObjectInput{
Bucket: &d.Bucket,
+1
View File
@@ -23,6 +23,7 @@ type Addition struct {
AddFilenameToDisposition bool `json:"add_filename_to_disposition" help:"Add filename to Content-Disposition header."`
EnableDirectUpload bool `json:"enable_direct_upload" default:"false"`
DirectUploadHost string `json:"direct_upload_host" required:"false"`
UserAgent string `json:"user_agent" required:"false" default:"" help:"Custom User-Agent for S3 requests."`
}
func init() {
+126 -10
View File
@@ -3,6 +3,7 @@ package s3
import (
"context"
"errors"
"fmt"
"net/http"
"net/url"
"path"
@@ -19,6 +20,13 @@ import (
log "github.com/sirupsen/logrus"
)
const (
maxCopyObjectSize int64 = 5 * 1000 * 1000 * 1000
defaultCopyPartSize int64 = 100 * 1024 * 1024
maxCopyPartSize int64 = 5 * 1024 * 1024 * 1024
maxCopyParts int64 = 10000
)
// do others that not defined in Driver interface
func (d *S3) initSession() error {
@@ -49,6 +57,11 @@ const (
func (d *S3) getClient(clientType int) *s3.S3 {
client := s3.New(d.Session)
if d.UserAgent != "" {
client.Handlers.Build.PushBack(func(r *request.Request) {
r.HTTPRequest.Header.Set("User-Agent", d.UserAgent)
})
}
if clientType == ClientTypeLink && d.CustomHost != "" {
client.Handlers.Build.PushBack(func(r *request.Request) {
if r.HTTPRequest.Method != http.MethodGet {
@@ -207,26 +220,129 @@ func (d *S3) listV2(dirPath string, args model.ListArgs) ([]model.Obj, error) {
return files, nil
}
func (d *S3) copy(ctx context.Context, src string, dst string, isDir bool) error {
func (d *S3) copy(ctx context.Context, src string, dst string, size int64, isDir bool) error {
if isDir {
return d.copyDir(ctx, src, dst)
}
return d.copyFile(ctx, src, dst)
return d.copyFile(ctx, src, dst, size)
}
func (d *S3) copyFile(ctx context.Context, src string, dst string) error {
func (d *S3) copyFile(ctx context.Context, src string, dst string, size int64) error {
srcKey := getKey(src, false)
dstKey := getKey(dst, false)
encodedKey := strings.ReplaceAll(url.PathEscape(d.Bucket+"/"+srcKey), "+", "%2B")
if size > maxCopyObjectSize {
return d.copyFileMultipart(ctx, srcKey, dstKey, encodedKey, size)
}
input := &s3.CopyObjectInput{
Bucket: &d.Bucket,
CopySource: aws.String(encodedKey),
Key: &dstKey,
}
_, err := d.client.CopyObject(input)
_, err := d.client.CopyObjectWithContext(ctx, input)
return err
}
func (d *S3) copyFileMultipart(ctx context.Context, srcKey, dstKey, encodedKey string, size int64) (err error) {
head, err := d.client.HeadObjectWithContext(ctx, &s3.HeadObjectInput{
Bucket: &d.Bucket,
Key: &srcKey,
})
if err != nil {
return err
}
if head.ContentLength != nil {
size = *head.ContentLength
}
partSize, err := getCopyPartSize(size)
if err != nil {
return err
}
createInput := &s3.CreateMultipartUploadInput{
Bucket: &d.Bucket,
Key: &dstKey,
CacheControl: head.CacheControl,
ContentDisposition: head.ContentDisposition,
ContentEncoding: head.ContentEncoding,
ContentLanguage: head.ContentLanguage,
ContentType: head.ContentType,
Metadata: head.Metadata,
WebsiteRedirectLocation: head.WebsiteRedirectLocation,
}
if head.Expires != nil {
if expires, parseErr := http.ParseTime(*head.Expires); parseErr == nil {
createInput.Expires = &expires
}
}
created, err := d.client.CreateMultipartUploadWithContext(ctx, createInput)
if err != nil {
return err
}
uploadID := aws.StringValue(created.UploadId)
if uploadID == "" {
return errors.New("create multipart upload returned an empty upload ID")
}
completed := false
defer func() {
if completed {
return
}
_, abortErr := d.client.AbortMultipartUploadWithContext(context.WithoutCancel(ctx), &s3.AbortMultipartUploadInput{
Bucket: &d.Bucket,
Key: &dstKey,
UploadId: &uploadID,
})
if abortErr != nil {
err = errors.Join(err, fmt.Errorf("failed to abort multipart copy: %w", abortErr))
}
}()
parts := make([]*s3.CompletedPart, 0, (size+partSize-1)/partSize)
for start, partNumber := int64(0), int64(1); start < size; start, partNumber = start+partSize, partNumber+1 {
end := min(start+partSize, size) - 1
copied, copyErr := d.client.UploadPartCopyWithContext(ctx, &s3.UploadPartCopyInput{
Bucket: &d.Bucket,
CopySource: &encodedKey,
CopySourceRange: aws.String(fmt.Sprintf("bytes=%d-%d", start, end)),
Key: &dstKey,
PartNumber: &partNumber,
UploadId: &uploadID,
})
if copyErr != nil {
return copyErr
}
if copied.CopyPartResult == nil || aws.StringValue(copied.CopyPartResult.ETag) == "" {
return fmt.Errorf("multipart copy part %d returned an empty ETag", partNumber)
}
parts = append(parts, &s3.CompletedPart{
ETag: copied.CopyPartResult.ETag,
PartNumber: &partNumber,
})
}
_, err = d.client.CompleteMultipartUploadWithContext(ctx, &s3.CompleteMultipartUploadInput{
Bucket: &d.Bucket,
Key: &dstKey,
UploadId: &uploadID,
MultipartUpload: &s3.CompletedMultipartUpload{
Parts: parts,
},
})
if err != nil {
return err
}
completed = true
return nil
}
func getCopyPartSize(size int64) (int64, error) {
partSize := max(defaultCopyPartSize, (size-1)/maxCopyParts+1)
if partSize > maxCopyPartSize {
return 0, fmt.Errorf("object size %d exceeds multipart copy limit", size)
}
return partSize, nil
}
func (d *S3) copyDir(ctx context.Context, src string, dst string) error {
objs, err := op.List(ctx, d, src, model.ListArgs{S3ShowPlaceholder: true})
if err != nil {
@@ -238,7 +354,7 @@ func (d *S3) copyDir(ctx context.Context, src string, dst string) error {
if obj.IsDir() {
err = d.copyDir(ctx, cSrc, cDst)
} else {
err = d.copyFile(ctx, cSrc, cDst)
err = d.copyFile(ctx, cSrc, cDst, obj.GetSize())
}
if err != nil {
return err
@@ -257,23 +373,23 @@ func (d *S3) removeDir(ctx context.Context, src string) error {
if obj.IsDir() {
err = d.removeDir(ctx, cSrc)
} else {
err = d.removeFile(cSrc)
err = d.removeFile(ctx, cSrc)
}
if err != nil {
return err
}
}
_ = d.removeFile(path.Join(src, getPlaceholderName(d.Placeholder)))
_ = d.removeFile(path.Join(src, d.Placeholder))
_ = d.removeFile(ctx, path.Join(src, getPlaceholderName(d.Placeholder)))
_ = d.removeFile(ctx, path.Join(src, d.Placeholder))
return nil
}
func (d *S3) removeFile(src string) error {
func (d *S3) removeFile(ctx context.Context, src string) error {
key := getKey(src, false)
input := &s3.DeleteObjectInput{
Bucket: &d.Bucket,
Key: &key,
}
_, err := d.client.DeleteObject(input)
_, err := d.client.DeleteObjectWithContext(ctx, input)
return err
}
+215
View File
@@ -0,0 +1,215 @@
package s3
import (
"context"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
awss3 "github.com/aws/aws-sdk-go/service/s3"
)
func TestCopyFileUsesCopyObjectAtLimit(t *testing.T) {
copyRequests := 0
d := newTestS3Driver(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPut || r.URL.Query().Get("uploadId") != "" {
t.Errorf("unexpected request: %s %s", r.Method, r.URL.String())
w.WriteHeader(http.StatusBadRequest)
return
}
copyRequests++
writeTestXML(t, w, `<CopyObjectResult><ETag>"copy"</ETag></CopyObjectResult>`)
})
if err := d.copyFile(context.Background(), "source+file", "destination", maxCopyObjectSize); err != nil {
t.Fatalf("copyFile: %v", err)
}
if copyRequests != 1 {
t.Fatalf("copy requests = %d, want 1", copyRequests)
}
}
func TestCopyFileUsesMultipartCopyAboveLimit(t *testing.T) {
size := maxCopyObjectSize + 1
wantParts := int((size + defaultCopyPartSize - 1) / defaultCopyPartSize)
ranges := make(map[int]string, wantParts)
completed := false
aborted := false
d := newTestS3Driver(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodHead:
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Cache-Control", "max-age=60")
w.Header().Set("Content-Disposition", "attachment")
w.Header().Set("Expires", "Wed, 21 Oct 2015 07:28:00 GMT")
w.Header().Set("X-Amz-Meta-Source", "preserved")
w.Header().Set("X-Amz-Website-Redirect-Location", "/redirect")
w.WriteHeader(http.StatusOK)
case r.Method == http.MethodPost && r.URL.Query().Has("uploads"):
if got := r.Header.Get("Cache-Control"); got != "max-age=60" {
t.Errorf("Cache-Control = %q, want %q", got, "max-age=60")
}
if got := r.Header.Get("Content-Disposition"); got != "attachment" {
t.Errorf("Content-Disposition = %q, want %q", got, "attachment")
}
if got := r.Header.Get("Content-Type"); got != "application/octet-stream" {
t.Errorf("Content-Type = %q, want %q", got, "application/octet-stream")
}
if got := r.Header.Get("Expires"); got != "Wed, 21 Oct 2015 07:28:00 GMT" {
t.Errorf("Expires = %q, want an unchanged HTTP date", got)
}
if got := r.Header.Get("X-Amz-Meta-Source"); got != "preserved" {
t.Errorf("metadata = %q, want %q", got, "preserved")
}
if got := r.Header.Get("X-Amz-Website-Redirect-Location"); got != "/redirect" {
t.Errorf("website redirect = %q, want %q", got, "/redirect")
}
writeTestXML(t, w, `<InitiateMultipartUploadResult><UploadId>upload-id</UploadId></InitiateMultipartUploadResult>`)
case r.Method == http.MethodPut && r.URL.Query().Get("uploadId") == "upload-id":
partNumber, err := strconv.Atoi(r.URL.Query().Get("partNumber"))
if err != nil {
t.Errorf("invalid part number: %v", err)
w.WriteHeader(http.StatusBadRequest)
return
}
if got := r.Header.Get("X-Amz-Copy-Source"); !strings.Contains(got, "source%2Bfile") {
t.Errorf("copy source = %q, want encoded source key", got)
}
ranges[partNumber] = r.Header.Get("X-Amz-Copy-Source-Range")
writeTestXML(t, w, fmt.Sprintf(`<CopyPartResult><ETag>"part-%d"</ETag></CopyPartResult>`, partNumber))
case r.Method == http.MethodPost && r.URL.Query().Get("uploadId") == "upload-id":
body, err := io.ReadAll(r.Body)
if err != nil {
t.Errorf("read complete body: %v", err)
}
if got := strings.Count(string(body), "<Part>"); got != wantParts {
t.Errorf("completed parts = %d, want %d", got, wantParts)
}
completed = true
writeTestXML(t, w, `<CompleteMultipartUploadResult><ETag>"complete"</ETag></CompleteMultipartUploadResult>`)
case r.Method == http.MethodDelete && r.URL.Query().Get("uploadId") == "upload-id":
aborted = true
w.WriteHeader(http.StatusNoContent)
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.String())
w.WriteHeader(http.StatusBadRequest)
}
})
if err := d.copyFile(context.Background(), "source+file", "destination", size); err != nil {
t.Fatalf("copyFile: %v", err)
}
if !completed {
t.Fatal("multipart upload was not completed")
}
if aborted {
t.Fatal("successful multipart upload was aborted")
}
if len(ranges) != wantParts {
t.Fatalf("copied parts = %d, want %d", len(ranges), wantParts)
}
if got := ranges[1]; got != fmt.Sprintf("bytes=0-%d", defaultCopyPartSize-1) {
t.Errorf("first range = %q", got)
}
lastStart := int64(wantParts-1) * defaultCopyPartSize
if got := ranges[wantParts]; got != fmt.Sprintf("bytes=%d-%d", lastStart, size-1) {
t.Errorf("last range = %q", got)
}
}
func TestCopyFileMultipartAbortsOnPartFailure(t *testing.T) {
size := maxCopyObjectSize + 1
aborted := false
completed := false
d := newTestS3Driver(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodHead:
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
w.WriteHeader(http.StatusOK)
case r.Method == http.MethodPost && r.URL.Query().Has("uploads"):
writeTestXML(t, w, `<InitiateMultipartUploadResult><UploadId>upload-id</UploadId></InitiateMultipartUploadResult>`)
case r.Method == http.MethodPut && r.URL.Query().Get("uploadId") == "upload-id":
w.WriteHeader(http.StatusInternalServerError)
writeTestXML(t, w, `<Error><Code>InternalError</Code><Message>copy failed</Message></Error>`)
case r.Method == http.MethodDelete && r.URL.Query().Get("uploadId") == "upload-id":
aborted = true
w.WriteHeader(http.StatusNoContent)
case r.Method == http.MethodPost && r.URL.Query().Get("uploadId") == "upload-id":
completed = true
w.WriteHeader(http.StatusOK)
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.String())
w.WriteHeader(http.StatusBadRequest)
}
})
if err := d.copyFile(context.Background(), "source", "destination", size); err == nil {
t.Fatal("copyFile returned nil error")
}
if !aborted {
t.Fatal("failed multipart upload was not aborted")
}
if completed {
t.Fatal("failed multipart upload was completed")
}
}
func TestGetCopyPartSize(t *testing.T) {
partSize, err := getCopyPartSize(defaultCopyPartSize * maxCopyParts)
if err != nil {
t.Fatalf("getCopyPartSize: %v", err)
}
if partSize != defaultCopyPartSize {
t.Fatalf("part size = %d, want %d", partSize, defaultCopyPartSize)
}
partSize, err = getCopyPartSize(defaultCopyPartSize*maxCopyParts + 1)
if err != nil {
t.Fatalf("getCopyPartSize: %v", err)
}
if partSize != defaultCopyPartSize+1 {
t.Fatalf("grown part size = %d, want %d", partSize, defaultCopyPartSize+1)
}
if _, err := getCopyPartSize(maxCopyPartSize*maxCopyParts + 1); err == nil {
t.Fatal("getCopyPartSize returned nil error for an oversized object")
}
}
func newTestS3Driver(t *testing.T, handler http.HandlerFunc) *S3 {
t.Helper()
server := httptest.NewServer(handler)
t.Cleanup(server.Close)
sess, err := session.NewSession(&aws.Config{
Credentials: credentials.NewStaticCredentials("access-key", "secret-key", ""),
Endpoint: aws.String(server.URL),
Region: aws.String("us-east-1"),
S3ForcePathStyle: aws.Bool(true),
MaxRetries: aws.Int(0),
})
if err != nil {
t.Fatalf("create AWS session: %v", err)
}
return &S3{
Addition: Addition{Bucket: "bucket"},
client: awss3.New(sess),
}
}
func writeTestXML(t *testing.T, w http.ResponseWriter, body string) {
t.Helper()
w.Header().Set("Content-Type", "application/xml")
if _, err := io.WriteString(w, body); err != nil {
t.Errorf("write response: %v", err)
}
}
-3
View File
@@ -46,9 +46,6 @@ func (d *Wps) Init(ctx context.Context) error {
if !resp.IsSuccess() {
return fmt.Errorf("failed to check login status, status code: %d, body: %s", resp.StatusCode(), resp.String())
}
if d.login.CompanyID == 0 {
return fmt.Errorf("wps company id is empty, please check business account login")
}
return nil
}
+110 -102
View File
@@ -1,12 +1,12 @@
module github.com/OpenListTeam/OpenList/v4
go 1.24.0
go 1.25.0
toolchain go1.24.13
toolchain go1.26.4
require (
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.1
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.2
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.22.0
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0
github.com/KarpelesLab/reflink v1.0.2
github.com/KirCute/zip v1.0.1
github.com/OpenListTeam/go-cache v0.1.0
@@ -14,9 +14,9 @@ require (
github.com/OpenListTeam/tache v0.2.2
github.com/OpenListTeam/times v0.1.0
github.com/OpenListTeam/wopan-sdk-go v0.1.5
github.com/ProtonMail/go-crypto v1.3.0
github.com/ProtonMail/gopenpgp/v2 v2.9.0
github.com/SheltonZhu/115driver v1.3.3
github.com/ProtonMail/go-crypto v1.4.1
github.com/ProtonMail/gopenpgp/v2 v2.10.0
github.com/SheltonZhu/115driver v1.3.5
github.com/aliyun/aliyun-oss-go-sdk v3.0.2+incompatible
github.com/antchfx/htmlquery v1.3.6
github.com/antchfx/xpath v1.3.6
@@ -25,24 +25,24 @@ require (
github.com/blevesearch/bleve/v2 v2.5.2
github.com/bmatcuk/doublestar/v4 v4.9.1
github.com/caarlos0/env/v9 v9.0.0
github.com/charmbracelet/bubbles v0.21.0
github.com/charmbracelet/bubbletea v1.3.6
github.com/charmbracelet/bubbles v0.21.1
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/city404/v6-public-rpc-proto/go v0.0.0-20240817070657-90f8e24b653e
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc
github.com/cloudsoda/go-smb2 v0.0.0-20260701064823-d8c5600d73b8
github.com/coreos/go-oidc v2.3.0+incompatible
github.com/deckarep/golang-set/v2 v2.8.0
github.com/deckarep/golang-set/v2 v2.9.0
github.com/dhowden/tag v0.0.0-20240417053706-3d75831295e8
github.com/disintegration/imaging v1.6.2
github.com/dlclark/regexp2 v1.11.5
github.com/dlclark/regexp2 v1.12.0
github.com/dustinxie/ecc v0.0.0-20210511000915-959544187564
github.com/fclairamb/ftpserverlib v0.26.1-0.20250709223522-4a925d79caf6
github.com/foxxorcat/mopan-sdk-go v0.1.6
github.com/foxxorcat/weiyun-sdk-go v0.1.4
github.com/gin-contrib/cors v1.7.6
github.com/gin-gonic/gin v1.10.1
github.com/gin-contrib/cors v1.7.7
github.com/gin-gonic/gin v1.12.0
github.com/glebarez/sqlite v1.11.0
github.com/go-resty/resty/v2 v2.16.5
github.com/go-resty/resty/v2 v2.17.2
github.com/go-webauthn/webauthn v0.13.4
github.com/golang-jwt/jwt/v4 v4.5.2
github.com/google/uuid v1.6.0
@@ -52,66 +52,71 @@ require (
github.com/henrybear327/go-proton-api v1.0.0
github.com/ipfs/go-ipfs-api v0.7.0
github.com/itsHenry35/gofakes3 v0.0.8
github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3
github.com/jlaffaye/ftp v0.2.1-0.20251026020404-6602e981a1bb
github.com/json-iterator/go v1.1.12
github.com/kdomanski/iso9660 v0.4.0
github.com/maruel/natural v1.1.1
github.com/maruel/natural v1.3.0
github.com/meilisearch/meilisearch-go v0.32.0
github.com/mholt/archives v0.1.3
github.com/mholt/archives v0.1.5
github.com/natefinch/lumberjack v2.0.0+incompatible
github.com/ncw/swift/v2 v2.0.4
github.com/ncw/swift/v2 v2.0.5
github.com/pkg/errors v0.9.1
github.com/pkg/sftp v1.13.9
github.com/pkg/sftp v1.13.10
github.com/pquerna/otp v1.5.0
github.com/quic-go/quic-go v0.54.1
github.com/rclone/rclone v1.70.3
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd
github.com/shirou/gopsutil/v4 v4.25.5
github.com/sirupsen/logrus v1.9.3
github.com/quic-go/quic-go v0.60.0
github.com/rclone/rclone v1.74.4
github.com/shirou/gopsutil/v4 v4.26.7
github.com/sirupsen/logrus v1.9.4
github.com/spf13/afero v1.15.0
github.com/spf13/cobra v1.10.2
github.com/stretchr/testify v1.11.1
github.com/t3rm1n4l/go-mega v0.0.0-20241213151442-a19cff0ec7b5
github.com/t3rm1n4l/go-mega v0.0.0-20260717075258-c6acd6a5bd04
github.com/tchap/go-patricia/v2 v2.3.3
github.com/u2takey/ffmpeg-go v0.5.0
github.com/upyun/go-sdk/v3 v3.0.4
github.com/winfsp/cgofuse v1.6.0
github.com/zzzhr1990/go-common-entity v0.0.0-20250202070650-1a200048f0d3
golang.org/x/crypto v0.46.0
golang.org/x/image v0.29.0
golang.org/x/net v0.48.0
golang.org/x/oauth2 v0.34.0
golang.org/x/time v0.14.0
golang.org/x/crypto v0.54.0
golang.org/x/image v0.43.0
golang.org/x/net v0.57.0
golang.org/x/oauth2 v0.36.0
golang.org/x/time v0.15.0
google.golang.org/appengine v1.6.8
gopkg.in/ldap.v3 v3.1.0
gorm.io/driver/mysql v1.5.7
gorm.io/driver/postgres v1.5.9
gorm.io/driver/mysql v1.6.0
gorm.io/driver/postgres v1.6.0
gorm.io/driver/sqlite v1.6.0
gorm.io/gorm v1.30.0
gorm.io/gorm v1.31.2
)
require (
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.1 // indirect
filippo.io/edwards25519 v1.1.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 // indirect
github.com/BurntSushi/toml v1.6.0 // indirect
github.com/ProtonMail/bcrypt v0.0.0-20211005172633-e235017c1baf // indirect
github.com/ProtonMail/gluon v0.17.1-0.20230724134000-308be39be96e // indirect
github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f // indirect
github.com/ProtonMail/go-srp v0.0.7 // indirect
github.com/PuerkitoBio/goquery v1.10.3 // indirect
github.com/PuerkitoBio/goquery v1.11.0 // indirect
github.com/RoaringBitmap/roaring/v2 v2.4.5 // indirect
github.com/adrg/xdg v0.5.3 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/charmbracelet/colorprofile v0.2.3-0.20250311203215-f60798e515dc // indirect
github.com/charmbracelet/x/cellbuf v0.0.13-0.20250311204145-2c3ea96c31dd // indirect
github.com/bytedance/gopkg v0.1.3 // indirect
github.com/charmbracelet/colorprofile v0.4.1 // indirect
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
github.com/clipperhouse/displaywidth v0.9.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.7.0 // indirect
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/cronokirby/saferith v0.33.0 // indirect
github.com/coreos/go-systemd/v22 v22.6.0 // indirect
github.com/cronokirby/saferith v0.33.1-0.20250226174546-1f11f94ce488 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/ebitengine/purego v0.8.4 // indirect
github.com/ebitengine/purego v0.10.2 // indirect
github.com/emersion/go-message v0.18.2 // indirect
github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff // indirect
github.com/geoffgarside/ber v1.2.0 // indirect
github.com/glebarez/go-sqlite v1.21.2 // indirect
github.com/goccy/go-yaml v1.19.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
@@ -120,19 +125,22 @@ require (
github.com/jcmturner/goidentity/v6 v6.0.1 // indirect
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/lanrat/extsort v1.0.2 // indirect
github.com/mattn/go-sqlite3 v1.14.22 // indirect
github.com/lanrat/extsort v1.4.2 // indirect
github.com/mikelolasagasti/xz v1.0.1 // indirect
github.com/minio/minlz v1.0.0 // indirect
github.com/minio/minlz v1.0.1 // indirect
github.com/minio/xxml v0.0.3 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/quic-go/qpack v0.5.1 // indirect
github.com/relvacode/iso8601 v1.6.0 // indirect
github.com/quic-go/qpack v0.6.0 // indirect
github.com/relvacode/iso8601 v1.7.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/stangelandcl/ppmd v0.1.1 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
go.uber.org/mock v0.5.0 // indirect
golang.org/x/exp v0.0.0-20250606033433-dcc06ee1d476 // indirect
golang.org/x/mod v0.30.0 // indirect
github.com/zeebo/blake3 v0.2.4 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
go.mongodb.org/mongo-driver v1.17.9 // indirect
go.mongodb.org/mongo-driver/v2 v2.5.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
gopkg.in/go-jose/go-jose.v2 v2.6.3 // indirect
modernc.org/libc v1.22.5 // indirect
modernc.org/mathutil v1.5.0 // indirect
@@ -141,19 +149,19 @@ require (
)
require (
github.com/OpenListTeam/115-sdk-go v0.2.4
github.com/STARRY-S/zip v0.2.1 // indirect
github.com/OpenListTeam/115-sdk-go v0.2.6
github.com/STARRY-S/zip v0.2.3 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/blevesearch/go-faiss v1.0.25 // indirect
github.com/blevesearch/zapx/v16 v16.2.4 // indirect
github.com/bodgit/plumbing v1.3.0 // indirect
github.com/bodgit/sevenzip v1.6.1
github.com/bodgit/sevenzip v1.6.5
github.com/bodgit/windows v1.0.1 // indirect
github.com/bytedance/sonic/loader v0.2.4 // indirect
github.com/charmbracelet/x/ansi v0.9.3 // indirect
github.com/charmbracelet/x/term v0.2.1 // indirect
github.com/cloudflare/circl v1.6.1 // indirect
github.com/cloudwego/base64x v0.1.5 // indirect
github.com/bytedance/sonic/loader v0.5.0 // indirect
github.com/charmbracelet/x/ansi v0.11.5 // indirect
github.com/charmbracelet/x/term v0.2.2 // indirect
github.com/cloudflare/circl v1.6.3 // indirect
github.com/cloudwego/base64x v0.1.6 // indirect
github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 // indirect
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
github.com/fclairamb/go-log v0.6.0 // indirect
@@ -162,14 +170,14 @@ require (
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hekmon/cunits/v2 v2.1.0 // indirect
github.com/ipfs/boxo v0.12.0 // indirect
github.com/jackc/puddle/v2 v2.2.1 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/pgzip v1.2.6 // indirect
github.com/matoous/go-nanoid/v2 v2.1.0 // indirect
github.com/microcosm-cc/bluemonday v1.0.27
github.com/nwaples/rardecode/v2 v2.1.1
github.com/sorairolake/lzip-go v0.3.5 // indirect
github.com/nwaples/rardecode/v2 v2.3.0
github.com/sorairolake/lzip-go v0.3.8 // indirect
github.com/taruti/bytepool v0.0.0-20160310082835-5e3a9ea56543 // indirect
github.com/ulikunitz/xz v0.5.12 // indirect
github.com/ulikunitz/xz v0.5.15 // indirect
go4.org v0.0.0-20260112195520-a5071408f32f
resty.dev/v3 v3.0.0-beta.2 // indirect
)
@@ -180,12 +188,12 @@ require (
github.com/abbot/go-http-auth v0.4.0 // indirect
github.com/aead/ecdh v0.2.0 // indirect
github.com/andreburgaud/crypt2go v1.8.0 // indirect
github.com/andybalholm/brotli v1.1.2-0.20250424173009-453214e765f3 // indirect
github.com/andybalholm/brotli v1.2.2 // indirect
github.com/axgle/mahonia v0.0.0-20180208002826-3358181d7394
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/benbjohnson/clock v1.3.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/bits-and-blooms/bitset v1.22.0 // indirect
github.com/bits-and-blooms/bitset v1.24.4 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/blevesearch/bleve_index_api v1.2.8 // indirect
github.com/blevesearch/geo v0.2.3 // indirect
@@ -202,25 +210,25 @@ require (
github.com/blevesearch/zapx/v13 v13.4.2 // indirect
github.com/blevesearch/zapx/v14 v14.4.2 // indirect
github.com/blevesearch/zapx/v15 v15.4.2 // indirect
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc // indirect
github.com/bytedance/sonic v1.13.3 // indirect
github.com/boombuler/barcode v1.1.0 // indirect
github.com/bytedance/sonic v1.15.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/coreos/go-semver v0.3.1 // indirect
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 // indirect
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.9 // indirect
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
github.com/gin-contrib/sse v1.1.0 // indirect
github.com/go-chi/chi/v5 v5.2.2 // indirect
github.com/go-chi/chi/v5 v5.2.5 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.26.0 // indirect
github.com/go-sql-driver/mysql v1.7.0 // indirect
github.com/go-playground/validator/v10 v10.30.1 // indirect
github.com/go-sql-driver/mysql v1.8.1 // indirect
github.com/go-webauthn/x v0.1.23 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/golang-jwt/jwt/v5 v5.2.3 // indirect
github.com/golang-jwt/jwt/v5 v5.3.1 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/go-tpm v0.9.5 // indirect
@@ -229,35 +237,36 @@ require (
github.com/hashicorp/go-version v1.6.0 // indirect
github.com/henrybear327/Proton-API-Bridge v1.0.0
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/ipfs/go-cid v0.5.0
github.com/ipfs/go-cid v0.6.2
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
github.com/jackc/pgx/v5 v5.5.5 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx/v5 v5.6.0 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/jzelinskie/whirlpool v0.0.0-20201016144138-0675e54bb004 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.10 // indirect
github.com/klauspost/compress v1.19.0 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/kr/fs v0.1.0 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/libp2p/go-buffer-pool v0.1.0 // indirect
github.com/libp2p/go-flow-metrics v0.1.0 // indirect
github.com/libp2p/go-libp2p v0.27.8 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35 // indirect
github.com/mailru/easyjson v0.9.0 // indirect
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
github.com/lufia/plan9stats v0.0.0-20260216142805-b3301c5f2a88 // indirect
github.com/mailru/easyjson v0.9.1 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/mattn/go-runewidth v0.0.22 // indirect
github.com/mattn/go-sqlite3 v1.14.22 // indirect
github.com/minio/sha256-simd v1.0.1 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/mr-tron/base58 v1.3.0 // indirect
github.com/mschoch/smat v0.2.0 // indirect
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
@@ -265,46 +274,45 @@ require (
github.com/multiformats/go-base32 v0.1.0 // indirect
github.com/multiformats/go-base36 v0.2.0 // indirect
github.com/multiformats/go-multiaddr v0.9.0 // indirect
github.com/multiformats/go-multibase v0.2.0 // indirect
github.com/multiformats/go-multibase v0.3.0 // indirect
github.com/multiformats/go-multicodec v0.9.0 // indirect
github.com/multiformats/go-multihash v0.2.3 // indirect
github.com/multiformats/go-multistream v0.4.1 // indirect
github.com/multiformats/go-varint v0.0.7 // indirect
github.com/multiformats/go-varint v0.1.0 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pierrec/lz4/v4 v4.1.27 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/pquerna/cachecontrol v0.1.0 // indirect
github.com/prometheus/client_golang v1.22.0 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.64.0 // indirect
github.com/prometheus/procfs v0.16.1 // indirect
github.com/rfjakob/eme v1.1.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.19.2 // indirect
github.com/rfjakob/eme v1.2.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46 // indirect
github.com/shabbyrobe/gocovmerge v0.0.0-20230507112040-c3350d9342df // indirect
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/tklauser/go-sysconf v0.3.15 // indirect
github.com/tklauser/numcpus v0.10.0 // indirect
github.com/tklauser/go-sysconf v0.3.16 // indirect
github.com/tklauser/numcpus v0.11.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/u2takey/go-utils v0.3.1 // indirect
github.com/ugorji/go/codec v1.3.0 // indirect
github.com/ugorji/go/codec v1.3.1 // indirect
github.com/x448/float16 v0.8.4 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.etcd.io/bbolt v1.4.0 // indirect
golang.org/x/arch v0.18.0 // indirect
golang.org/x/sync v0.19.0
golang.org/x/sys v0.40.0
golang.org/x/term v0.38.0 // indirect
golang.org/x/text v0.32.0
golang.org/x/tools v0.39.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b // indirect
google.golang.org/grpc v1.78.0
go.etcd.io/bbolt v1.4.3 // indirect
golang.org/x/arch v0.23.0 // indirect
golang.org/x/sync v0.22.0
golang.org/x/sys v0.47.0
golang.org/x/term v0.45.0 // indirect
golang.org/x/text v0.40.0
golang.org/x/tools v0.47.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478 // indirect
google.golang.org/grpc v1.82.0
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d // indirect
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/blake3 v1.1.7 // indirect
)
@@ -315,4 +323,4 @@ replace github.com/cronokirby/saferith => github.com/Da3zKi7/saferith v0.33.0-fi
// replace github.com/OpenListTeam/115-sdk-go => ../../OpenListTeam/115-sdk-go
replace github.com/KarpelesLab/reflink => github.com/OpenListTeam/reflink v0.0.0-20260520031008-ed3c0dbe8009
replace github.com/KarpelesLab/reflink => github.com/OpenListTeam/reflink v0.0.0-20260701021214-78760eaeafef
+269 -156
View File
@@ -5,16 +5,30 @@ cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIi
cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.1 h1:Wc1ml6QlJs2BHQ/9Bqu1jiyggbsSjramq2oUmp5WeIo=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.18.1/go.mod h1:Ot/6aikWnKWi4l9QB7qVSwa8iMphQNqkWALMoNT3rzM=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0 h1:fou+2+WFTib47nS+nz/ozhEBnvU96bKHy6LjRsY4E28=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.21.0/go.mod h1:t76Ruy8AHvUAC8GfMWJMa0ElSbuIcO03NLpynfbgsPA=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.22.0 h1:aokoqcHvaGjiM3VpjKDfMMnF/8epJ+Q1HLJ7CudztqE=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.22.0/go.mod h1:/WYEx9pcM9Y+Dd/APJaNlSvVSvzl54rrMdZT5+Oi2LM=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.10.1 h1:B+blDbyVIG3WaikNxPnhPiJ1MThR03b3vKGtER95TP4=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.10.1/go.mod h1:JdM5psgjfBf5fo2uWOZhflPWyDBZ/O/CNAH9CtsuZE4=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.1 h1:FPKJS1T+clwv+OLGt13a8UjqeRuh0O4SJ3lUriThc+4=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.1/go.mod h1:j2chePtV91HrC22tGoRX3sGY42uF13WzmmV80/OdVAA=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2 h1:9iefClla7iYpfYWdzPCRDozdmndjTm8DXdpCzPajMgA=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.11.2/go.mod h1:XtLgD3ZD34DAaVIIAyG3objl5DynM3CQ/vMcbBNJZGI=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 h1:fhqpLE3UEXi9lPaBRpQ6XuRW0nU7hgg4zlmZZa+a9q4=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0/go.mod h1:7dCRMLwisfRH3dBupKeNCioWYUZ4SS09Z14H+7i8ZoY=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1 h1:/Zt+cDPnpC3OVDm/JKLOs7M2DKmLRIIp3XIx9pHHiig=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.8.1/go.mod h1:Ng3urmn6dYe8gnbCMoHHVl5APYz2txho3koEkV2o2HA=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.2 h1:FwladfywkNirM+FZYLBR2kBz5C8Tg0fw5w5Y7meRXWI=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.2/go.mod h1:vv5Ad0RrIoT1lJFdWBZwt4mB1+j+V8DUroixmKDTCdk=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4 h1:jWQK1GI+LeGGUKBADtcH2rRqPxYB1Ljwms5gFA2LqrM=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4/go.mod h1:8mwH4klAm9DUgR2EEHyEEAQlRDvLPyg5fQry3y+cDew=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0 h1:irsmOWwkp0KCTTNS5e2hdFeIvSQClQo2No3IaNmL3Vw=
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.8.0/go.mod h1:GWcBkQj3MqN7ozHKLaCCAuNLiXoIGv2RtanfAwSjY/Y=
github.com/AzureAD/microsoft-authentication-library-for-go v1.4.2 h1:oygO0locgZJe7PpYPXT5A29ZkwJaPqcva7BVeemZOZs=
github.com/AzureAD/microsoft-authentication-library-for-go v1.4.2/go.mod h1:wP83P5OoQ5p6ip3ScPr0BAq0BvuPAvacpEuSzyouqAI=
github.com/BurntSushi/toml v1.6.0 h1:dRaEfpa2VI55EwlIW72hMRHdWouJeRF7TPYhI+AUQjk=
@@ -29,12 +43,16 @@ github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd h1:nzE1YQBdx1bq9
github.com/Max-Sum/base32768 v0.0.0-20230304063302-18e6ce5945fd/go.mod h1:C8yoIfvESpM3GD07OCHU7fqI7lhwyZ2Td1rbNbTAhnc=
github.com/OpenListTeam/115-sdk-go v0.2.4 h1:dVoEz3Pm996n/ZCuRckQvz36w938uNHjPrS7E/SVpBM=
github.com/OpenListTeam/115-sdk-go v0.2.4/go.mod h1:cfvitk2lwe6036iNi2h+iNxwxWDifKZsSvNtrur5BqU=
github.com/OpenListTeam/115-sdk-go v0.2.5 h1:E4O7GZmEXlGnouZ/e9//xU7NepT9JJkWtnF3O/VSb08=
github.com/OpenListTeam/115-sdk-go v0.2.5/go.mod h1:cfvitk2lwe6036iNi2h+iNxwxWDifKZsSvNtrur5BqU=
github.com/OpenListTeam/115-sdk-go v0.2.6 h1:ehXyStvncvn4qRBuknor3kyGZtUmHc0+stjF2ByVdFo=
github.com/OpenListTeam/115-sdk-go v0.2.6/go.mod h1:cfvitk2lwe6036iNi2h+iNxwxWDifKZsSvNtrur5BqU=
github.com/OpenListTeam/go-cache v0.1.0 h1:eV2+FCP+rt+E4OCJqLUW7wGccWZNJMV0NNkh+uChbAI=
github.com/OpenListTeam/go-cache v0.1.0/go.mod h1:AHWjKhNK3LE4rorVdKyEALDHoeMnP8SjiNyfVlB+Pz4=
github.com/OpenListTeam/gsync v0.1.0 h1:ywzGybOvA3lW8K1BUjKZ2IUlT2FSlzPO4DOazfYXjcs=
github.com/OpenListTeam/gsync v0.1.0/go.mod h1:h/Rvv9aX/6CdW/7B8di3xK3xNV8dUg45Fehrd/ksZ9s=
github.com/OpenListTeam/reflink v0.0.0-20260520031008-ed3c0dbe8009 h1:qLqJPr/FAsZTJiqy65JKKuFJP0V9pRVtSaIE0kqaQ8w=
github.com/OpenListTeam/reflink v0.0.0-20260520031008-ed3c0dbe8009/go.mod h1:WGkTOKNjd1FsJKBw3mu4JvrPEDJyJJ+JPtxBkbPoCok=
github.com/OpenListTeam/reflink v0.0.0-20260701021214-78760eaeafef h1:67uGHancMF/abMrnkc8abVUWQiG73Wk5d8CKt3RzkFo=
github.com/OpenListTeam/reflink v0.0.0-20260701021214-78760eaeafef/go.mod h1:WGkTOKNjd1FsJKBw3mu4JvrPEDJyJJ+JPtxBkbPoCok=
github.com/OpenListTeam/sftpd-openlist v1.0.1 h1:j4S3iPFOpnXCUKRPS7uCT4mF2VCl34GyqvH6lqwnkUU=
github.com/OpenListTeam/sftpd-openlist v1.0.1/go.mod h1:uO/wKnbvbdq3rBLmClMTZXuCnw7XW4wlAq4dZe91a40=
github.com/OpenListTeam/tache v0.2.2 h1:CWFn6sr1AIYaEjC8ONdKs+LrxHyuErheenAjEqRhh4k=
@@ -49,24 +67,30 @@ github.com/ProtonMail/bcrypt v0.0.0-20211005172633-e235017c1baf/go.mod h1:o0ESU9
github.com/ProtonMail/gluon v0.17.1-0.20230724134000-308be39be96e h1:lCsqUUACrcMC83lg5rTo9Y0PnPItE61JSfvMyIcANwk=
github.com/ProtonMail/gluon v0.17.1-0.20230724134000-308be39be96e/go.mod h1:Og5/Dz1MiGpCJn51XujZwxiLG7WzvvjE5PRpZBQmAHo=
github.com/ProtonMail/go-crypto v0.0.0-20230321155629-9a39f2531310/go.mod h1:8TI4H3IbrackdNgv+92dI+rhpCaLqM0IfpgCgenFvRE=
github.com/ProtonMail/go-crypto v1.3.0 h1:ILq8+Sf5If5DCpHQp4PbZdS1J7HDFRXz/+xKBiRGFrw=
github.com/ProtonMail/go-crypto v1.3.0/go.mod h1:9whxjD8Rbs29b4XWbB8irEcE8KHMqaR2e7GWU1R+/PE=
github.com/ProtonMail/go-crypto v1.4.1 h1:9RfcZHqEQUvP8RzecWEUafnZVtEvrBVL9BiF67IQOfM=
github.com/ProtonMail/go-crypto v1.4.1/go.mod h1:e1OaTyu5SYVrO9gKOEhTc+5UcXtTUa+P3uLudwcgPqo=
github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f h1:tCbYj7/299ekTTXpdwKYF8eBlsYsDVoggDAuAjoK66k=
github.com/ProtonMail/go-mime v0.0.0-20230322103455-7d82a3887f2f/go.mod h1:gcr0kNtGBqin9zDW9GOHcVntrwnjrK+qdJ06mWYBybw=
github.com/ProtonMail/go-srp v0.0.7 h1:Sos3Qk+th4tQR64vsxGIxYpN3rdnG9Wf9K4ZloC1JrI=
github.com/ProtonMail/go-srp v0.0.7/go.mod h1:giCp+7qRnMIcCvI6V6U3S1lDDXDQYx2ewJ6F/9wdlJk=
github.com/ProtonMail/gopenpgp/v2 v2.9.0 h1:ruLzBmwe4dR1hdnrsEJ/S7psSBmV15gFttFUPP/+/kE=
github.com/ProtonMail/gopenpgp/v2 v2.9.0/go.mod h1:IldDyh9Hv1ZCCYatTuuEt1XZJ0OPjxLpTarDfglih7s=
github.com/ProtonMail/gopenpgp/v2 v2.10.0 h1:llCzLvntC9+iH+if/na4AgKTef/Zm4vpaRrR3+JdKvo=
github.com/ProtonMail/gopenpgp/v2 v2.10.0/go.mod h1:dc0h9Pg3ftfN0U4pfRzujilfh61A2R52wgMkZWcWm2I=
github.com/PuerkitoBio/goquery v1.10.3 h1:pFYcNSqHxBD06Fpj/KsbStFRsgRATgnf3LeXiUkhzPo=
github.com/PuerkitoBio/goquery v1.10.3/go.mod h1:tMUX0zDMHXYlAQk6p35XxQMqMweEKB7iK7iLNd4RH4Y=
github.com/PuerkitoBio/goquery v1.11.0 h1:jZ7pwMQXIITcUXNH83LLk+txlaEy6NVOfTuP43xxfqw=
github.com/PuerkitoBio/goquery v1.11.0/go.mod h1:wQHgxUOU3JGuj3oD/QFfxUdlzW6xPHfqyHre6VMY4DQ=
github.com/RoaringBitmap/roaring/v2 v2.4.5 h1:uGrrMreGjvAtTBobc0g5IrW1D5ldxDQYe2JW2gggRdg=
github.com/RoaringBitmap/roaring/v2 v2.4.5/go.mod h1:FiJcsfkGje/nZBZgCu0ZxCPOKD/hVXDS2dXi7/eUFE0=
github.com/STARRY-S/zip v0.2.1 h1:pWBd4tuSGm3wtpoqRZZ2EAwOmcHK6XFf7bU9qcJXyFg=
github.com/STARRY-S/zip v0.2.1/go.mod h1:xNvshLODWtC4EJ702g7cTYn13G53o1+X9BWnPFpcWV4=
github.com/SheltonZhu/115driver v1.3.3 h1:Bqs86D2MziYPgIOuOJF+HzG4d7GBr71ZhSCrs/U17UU=
github.com/SheltonZhu/115driver v1.3.3/go.mod h1:OujS7azslg1/bn85sPSHnNsp4/WBI9/TiijtZL9kuSQ=
github.com/STARRY-S/zip v0.2.3 h1:luE4dMvRPDOWQdeDdUxUoZkzUIpTccdKdhHHsQJ1fm4=
github.com/STARRY-S/zip v0.2.3/go.mod h1:lqJ9JdeRipyOQJrYSOtpNAiaesFO6zVDsE8GIGFaoSk=
github.com/SheltonZhu/115driver v1.3.4 h1:joUFKN591n7KiTEGnBw0ZHjPzhl3G1PpZrwv2YHhoC8=
github.com/SheltonZhu/115driver v1.3.4/go.mod h1:OujS7azslg1/bn85sPSHnNsp4/WBI9/TiijtZL9kuSQ=
github.com/SheltonZhu/115driver v1.3.5 h1:NlJA61RskdwHhIZyg4519696lCk1n4aid0lKARsMwFg=
github.com/SheltonZhu/115driver v1.3.5/go.mod h1:ag0B8Vs6TPFQjTNFLVVxA7dINPt4q+WR47+JIPJhmTk=
github.com/abbot/go-http-auth v0.4.0 h1:QjmvZ5gSC7jm3Zg54DqWE/T5m1t2AfDu6QlXJT0EVT0=
github.com/abbot/go-http-auth v0.4.0/go.mod h1:Cz6ARTIzApMJDzh5bRMSUou6UMSp0IEXg9km/ci7TJM=
github.com/adrg/xdg v0.5.3 h1:xRnxJXne7+oWDatRhR1JLnvuccuIeCoBu2rtuLqQB78=
github.com/adrg/xdg v0.5.3/go.mod h1:nlTsY+NNiCBGCK2tpm09vRqfVzrc2fLmXGpBLF0zlTQ=
github.com/aead/ecdh v0.2.0 h1:pYop54xVaq/CEREFEcukHRZfTdjiWvYIsZDXXrBapQQ=
github.com/aead/ecdh v0.2.0/go.mod h1:a9HHtXuSo8J1Js1MwLQx2mBhkXMT6YwUmVVEY4tTB8U=
github.com/aliyun/aliyun-oss-go-sdk v3.0.2+incompatible h1:8psS8a+wKfiLt1iVDX79F7Y6wUM49Lcha2FMXt4UM8g=
@@ -74,8 +98,8 @@ github.com/aliyun/aliyun-oss-go-sdk v3.0.2+incompatible/go.mod h1:T/Aws4fEfogEE9
github.com/andreburgaud/crypt2go v1.8.0 h1:J73vGTb1P6XL69SSuumbKs0DWn3ulbl9L92ZXBjw6pc=
github.com/andreburgaud/crypt2go v1.8.0/go.mod h1:L5nfShQ91W78hOWhUH2tlGRPO+POAPJAF5fKOLB9SXg=
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/andybalholm/brotli v1.1.2-0.20250424173009-453214e765f3 h1:8PmGpDEZl9yDpcdEr6Odf23feCxK3LNUNMxjXg41pZQ=
github.com/andybalholm/brotli v1.1.2-0.20250424173009-453214e765f3/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/andybalholm/brotli v1.2.2 h1:HzTuoo2ErYQqf5qvcJInB8uvqSVxRttzkFexPWtnceM=
github.com/andybalholm/brotli v1.2.2/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/antchfx/htmlquery v1.3.6 h1:RNHHL7YehO5XdO8IM8CynwLKONwRHWkrghbYhQIk9ag=
@@ -117,8 +141,8 @@ github.com/axgle/mahonia v0.0.0-20180208002826-3358181d7394 h1:OYA+5W64v3OgClL+I
github.com/axgle/mahonia v0.0.0-20180208002826-3358181d7394/go.mod h1:Q8n74mJTIgjX4RBBcHnJ05h//6/k6foqmgE45jTQtxg=
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
github.com/aymanbagabas/go-udiff v0.2.0 h1:TK0fH4MteXUDspT88n8CKzvK0X9O2xu9yQjWpi6yML8=
github.com/aymanbagabas/go-udiff v0.2.0/go.mod h1:RE4Ex0qsGkTAJoQdQQCA0uG+nAzJO/pI/QwceO5fgrA=
github.com/aymanbagabas/go-udiff v0.3.1 h1:LV+qyBQ2pqe0u42ZsUEtPiCaUoqgA9gYRDs3vj1nolY=
github.com/aymanbagabas/go-udiff v0.3.1/go.mod h1:G0fsKmG+P6ylD0r6N/KgQD/nWzgfnl8ZBcNLgcbrw8E=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/benbjohnson/clock v1.3.0 h1:ip6w0uFQkncKQ979AypyG0ER7mqUSBdKLOgAle/AT8A=
@@ -126,8 +150,8 @@ github.com/benbjohnson/clock v1.3.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZx
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/bits-and-blooms/bitset v1.12.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
github.com/bits-and-blooms/bitset v1.22.0 h1:Tquv9S8+SGaS3EhyA+up3FXzmkhxPGjQQCkcs2uw7w4=
github.com/bits-and-blooms/bitset v1.22.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
github.com/bits-and-blooms/bitset v1.24.4 h1:95H15Og1clikBrKr/DuzMXkQzECs1M6hhoGXLwLQOZE=
github.com/bits-and-blooms/bitset v1.24.4/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
github.com/blevesearch/bleve/v2 v2.5.2 h1:Ab0r0MODV2C5A6BEL87GqLBySqp/s9xFgceCju6BQk8=
@@ -170,60 +194,74 @@ github.com/bmatcuk/doublestar/v4 v4.9.1 h1:X8jg9rRZmJd4yRy7ZeNDRnM+T3ZfHv15JiBJ/
github.com/bmatcuk/doublestar/v4 v4.9.1/go.mod h1:xBQ8jztBU6kakFMg+8WGxn0c6z1fTSPVIjEY1Wr7jzc=
github.com/bodgit/plumbing v1.3.0 h1:pf9Itz1JOQgn7vEOE7v7nlEfBykYqvUYioC61TwWCFU=
github.com/bodgit/plumbing v1.3.0/go.mod h1:JOTb4XiRu5xfnmdnDJo6GmSbSbtSyufrsyZFByMtKEs=
github.com/bodgit/sevenzip v1.6.1 h1:kikg2pUMYC9ljU7W9SaqHXhym5HyKm8/M/jd31fYan4=
github.com/bodgit/sevenzip v1.6.1/go.mod h1:GVoYQbEVbOGT8n2pfqCIMRUaRjQ8F9oSqoBEqZh5fQ8=
github.com/bodgit/sevenzip v1.6.5 h1:7H7BxgmeX0j6UX42lH+KXQ92WgMQJ49DoocFdfHbCng=
github.com/bodgit/sevenzip v1.6.5/go.mod h1:GhuB6Lq1xCpP1sps+horjZ8lgiKPJcy2zUX3prla9wc=
github.com/bodgit/windows v1.0.1 h1:tF7K6KOluPYygXa3Z2594zxlkbKPAOvqr97etrGNIz4=
github.com/bodgit/windows v1.0.1/go.mod h1:a6JLwrB4KrTR5hBpp8FI9/9W9jJfeQ2h4XDXU74ZCdM=
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc h1:biVzkmvwrH8WK8raXaxBx6fRVTlJILwEwQGL1I/ByEI=
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/boombuler/barcode v1.1.0 h1:ChaYjBR63fr4LFyGn8E8nt7dBSt3MiU3zMOZqFvVkHo=
github.com/boombuler/barcode v1.1.0/go.mod h1:paBWMcWSl3LHKBqUq+rly7CNSldXjb2rDl3JlRe0mD8=
github.com/bradenaw/juniper v0.15.3 h1:RHIAMEDTpvmzV1wg1jMAHGOoI2oJUSPx3lxRldXnFGo=
github.com/bradenaw/juniper v0.15.3/go.mod h1:UX4FX57kVSaDp4TPqvSjkAAewmRFAfXf27BOs5z9dq8=
github.com/bwesterb/go-ristretto v1.2.0/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
github.com/bytedance/sonic v1.13.3 h1:MS8gmaH16Gtirygw7jV91pDCN33NyMrPbN7qiYhEsF0=
github.com/bytedance/sonic v1.13.3/go.mod h1:o68xyaF9u2gvVBuGHPlUVCy+ZfmNNO5ETf1+KgkJhz4=
github.com/bytedance/sonic/loader v0.1.1/go.mod h1:ncP89zfokxS5LZrJxl5z0UJcsk4M4yY2JpfqGeCtNLU=
github.com/bytedance/sonic/loader v0.2.4 h1:ZWCw4stuXUsn1/+zQDqeE7JKP+QO47tz7QCNan80NzY=
github.com/bytedance/sonic/loader v0.2.4/go.mod h1:N8A3vUdtUebEY2/VQC0MyhYeKUFosQU6FxH2JmUe6VI=
github.com/bytedance/gopkg v0.1.3 h1:TPBSwH8RsouGCBcMBktLt1AymVo2TVsBVCY4b6TnZ/M=
github.com/bytedance/gopkg v0.1.3/go.mod h1:576VvJ+eJgyCzdjS+c4+77QF3p7ubbtiKARP3TxducM=
github.com/bytedance/sonic v1.15.0 h1:/PXeWFaR5ElNcVE84U0dOHjiMHQOwNIx3K4ymzh/uSE=
github.com/bytedance/sonic v1.15.0/go.mod h1:tFkWrPz0/CUCLEF4ri4UkHekCIcdnkqXw9VduqpJh0k=
github.com/bytedance/sonic/loader v0.5.0 h1:gXH3KVnatgY7loH5/TkeVyXPfESoqSBSBEiDd5VjlgE=
github.com/bytedance/sonic/loader v0.5.0/go.mod h1:AR4NYCk5DdzZizZ5djGqQ92eEhCCcdf5x77udYiSJRo=
github.com/caarlos0/env/v9 v9.0.0 h1:SI6JNsOA+y5gj9njpgybykATIylrRMklbs5ch6wO6pc=
github.com/caarlos0/env/v9 v9.0.0/go.mod h1:ye5mlCVMYh6tZ+vCgrs/B95sj88cg5Tlnc0XIzgZ020=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/charmbracelet/bubbles v0.21.0 h1:9TdC97SdRVg/1aaXNVWfFH3nnLAwOXr8Fn6u6mfQdFs=
github.com/charmbracelet/bubbles v0.21.0/go.mod h1:HF+v6QUR4HkEpz62dx7ym2xc71/KBHg+zKwJtMw+qtg=
github.com/charmbracelet/bubbletea v1.3.6 h1:VkHIxPJQeDt0aFJIsVxw8BQdh/F/L2KKZGsK6et5taU=
github.com/charmbracelet/bubbletea v1.3.6/go.mod h1:oQD9VCRQFF8KplacJLo28/jofOI2ToOfGYeFgBBxHOc=
github.com/charmbracelet/colorprofile v0.2.3-0.20250311203215-f60798e515dc h1:4pZI35227imm7yK2bGPcfpFEmuY1gc2YSTShr4iJBfs=
github.com/charmbracelet/colorprofile v0.2.3-0.20250311203215-f60798e515dc/go.mod h1:X4/0JoqgTIPSFcRA/P6INZzIuyqdFY5rm8tb41s9okk=
github.com/charmbracelet/bubbles v0.21.1 h1:nj0decPiixaZeL9diI4uzzQTkkz1kYY8+jgzCZXSmW0=
github.com/charmbracelet/bubbles v0.21.1/go.mod h1:HHvIYRCpbkCJw2yo0vNX1O5loCwSr9/mWS8GYSg50Sk=
github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
github.com/charmbracelet/colorprofile v0.4.1 h1:a1lO03qTrSIRaK8c3JRxJDZOvhvIeSco3ej+ngLk1kk=
github.com/charmbracelet/colorprofile v0.4.1/go.mod h1:U1d9Dljmdf9DLegaJ0nGZNJvoXAhayhmidOdcBwAvKk=
github.com/charmbracelet/lipgloss v1.1.0 h1:vYXsiLHVkK7fp74RkV7b2kq9+zDLoEU4MZoFqR/noCY=
github.com/charmbracelet/lipgloss v1.1.0/go.mod h1:/6Q8FR2o+kj8rz4Dq0zQc3vYf7X+B0binUUBwA0aL30=
github.com/charmbracelet/x/ansi v0.9.3 h1:BXt5DHS/MKF+LjuK4huWrC6NCvHtexww7dMayh6GXd0=
github.com/charmbracelet/x/ansi v0.9.3/go.mod h1:3RQDQ6lDnROptfpWuUVIUG64bD2g2BgntdxH0Ya5TeE=
github.com/charmbracelet/x/cellbuf v0.0.13-0.20250311204145-2c3ea96c31dd h1:vy0GVL4jeHEwG5YOXDmi86oYw2yuYUGqz6a8sLwg0X8=
github.com/charmbracelet/x/cellbuf v0.0.13-0.20250311204145-2c3ea96c31dd/go.mod h1:xe0nKWGd3eJgtqZRaN9RjMtK7xUYchjzPr7q6kcvCCs=
github.com/charmbracelet/x/ansi v0.11.5 h1:NBWeBpj/lJPE3Q5l+Lusa4+mH6v7487OP8K0r1IhRg4=
github.com/charmbracelet/x/ansi v0.11.5/go.mod h1:2JNYLgQUsyqaiLovhU2Rv/pb8r6ydXKS3NIttu3VGZQ=
github.com/charmbracelet/x/cellbuf v0.0.15 h1:ur3pZy0o6z/R7EylET877CBxaiE1Sp1GMxoFPAIztPI=
github.com/charmbracelet/x/cellbuf v0.0.15/go.mod h1:J1YVbR7MUuEGIFPCaaZ96KDl5NoS0DAWkskup+mOY+Q=
github.com/charmbracelet/x/exp/golden v0.0.0-20241011142426-46044092ad91 h1:payRxjMjKgx2PaCWLZ4p3ro9y97+TVLZNaRZgJwSVDQ=
github.com/charmbracelet/x/exp/golden v0.0.0-20241011142426-46044092ad91/go.mod h1:wDlXFlCrmJ8J+swcL/MnGUuYnqgQdW9rhSD61oNMb6U=
github.com/charmbracelet/x/term v0.2.1 h1:AQeHeLZ1OqSXhrAWpYUtZyX1T3zVxfpZuEQMIQaGIAQ=
github.com/charmbracelet/x/term v0.2.1/go.mod h1:oQ4enTYFV7QN4m0i9mzHrViD7TQKvNEEkHUMCmsxdUg=
github.com/charmbracelet/x/term v0.2.2 h1:xVRT/S2ZcKdhhOuSP4t5cLi5o+JxklsoEObBSgfgZRk=
github.com/charmbracelet/x/term v0.2.2/go.mod h1:kF8CY5RddLWrsgVwpw4kAa6TESp6EB5y3uxGLeCqzAI=
github.com/cheekybits/is v0.0.0-20150225183255-68e9c0620927 h1:SKI1/fuSdodxmNNyVBR8d7X/HuLnRpvvFO0AgyQk764=
github.com/cheekybits/is v0.0.0-20150225183255-68e9c0620927/go.mod h1:h/aW8ynjgkuj+NQRlZcDbAbM1ORAbXjXX77sX7T289U=
github.com/city404/v6-public-rpc-proto/go v0.0.0-20240817070657-90f8e24b653e h1:GLC8iDDcbt1H8+RkNao2nRGjyNTIo81e1rAJT9/uWYA=
github.com/city404/v6-public-rpc-proto/go v0.0.0-20240817070657-90f8e24b653e/go.mod h1:ln9Whp+wVY/FTbn2SK0ag+SKD2fC0yQCF/Lqowc1LmU=
github.com/clipperhouse/displaywidth v0.9.0 h1:Qb4KOhYwRiN3viMv1v/3cTBlz3AcAZX3+y9OLhMtAtA=
github.com/clipperhouse/displaywidth v0.9.0/go.mod h1:aCAAqTlh4GIVkhQnJpbL0T/WfcrJXHcj8C0yjYcjOZA=
github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs=
github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
github.com/clipperhouse/uax29/v2 v2.5.0 h1:x7T0T4eTHDONxFJsL94uKNKPHrclyFI0lm7+w94cO8U=
github.com/clipperhouse/uax29/v2 v2.5.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk=
github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM=
github.com/cloudflare/circl v1.1.0/go.mod h1:prBCrKB9DV4poKZY1l9zBXg2QJY7mvgRvtMxxK7fi4I=
github.com/cloudflare/circl v1.6.1 h1:zqIqSPIndyBh1bjLVVDHMPpVKqp8Su/V+6MeDzzQBQ0=
github.com/cloudflare/circl v1.6.1/go.mod h1:uddAzsPgqdMAYatqJ0lsjX1oECcQLIlRpzZh3pJrofs=
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc h1:t8YjNUCt1DimB4HCIXBztwWMhgxr5yG5/YaRl9Afdfg=
github.com/cloudsoda/go-smb2 v0.0.0-20250228001242-d4c70e6251cc/go.mod h1:CgWpFCFWzzEA5hVkhAc6DZZzGd3czx+BblvOzjmg6KA=
github.com/cloudflare/circl v1.6.2 h1:hL7VBpHHKzrV5WTfHCaBsgx/HGbBYlgrwvNXEVDYYsQ=
github.com/cloudflare/circl v1.6.2/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4=
github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg8=
github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4=
github.com/cloudsoda/go-smb2 v0.0.0-20260701064823-d8c5600d73b8 h1:+KC2I+emnT6ccWmo7IHbtchGc/vFqAxUuovo2eUgMuM=
github.com/cloudsoda/go-smb2 v0.0.0-20260701064823-d8c5600d73b8/go.mod h1:1pQXB0vAlzRlqcY7LYKOOZMw0wKfJPFxTLsJRF2Gswo=
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc h1:0xCWmFKBmarCqqqLeM7jFBSw/Or81UEElFqO8MY+GDs=
github.com/cloudsoda/sddl v0.0.0-20250224235906-926454e91efc/go.mod h1:uvR42Hb/t52HQd7x5/ZLzZEK8oihrFpgnodIJ1vte2E=
github.com/cloudwego/base64x v0.1.5 h1:XPciSp1xaq2VCSt6lF0phncD4koWyULpl5bUxbfCyP4=
github.com/cloudwego/base64x v0.1.5/go.mod h1:0zlkT4Wn5C6NdauXdJRhSKRlJvmclQ1hhJgA0rcu/8w=
github.com/cloudwego/iasm v0.2.0/go.mod h1:8rXZaNYT2n95jn+zTI1sDr+IgcD2GVs0nlbbQPiEFhY=
github.com/cloudwego/base64x v0.1.6 h1:t11wG9AECkCDk5fMSoxmufanudBtJ+/HemLstXDLI2M=
github.com/cloudwego/base64x v0.1.6/go.mod h1:OFcloc187FXDaYHvrNIjxSe8ncn0OOM8gEHfghB2IPU=
github.com/coreos/go-oidc v2.3.0+incompatible h1:+5vEsrgprdLjjQ9FzIKAzQz1wwPD+83hQRfUIPh7rO0=
github.com/coreos/go-oidc v2.3.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc=
github.com/coreos/go-semver v0.3.1 h1:yi21YpKnrx1gt5R+la8n5WgS0kCrsPp33dmEyHReZr4=
github.com/coreos/go-semver v0.3.1/go.mod h1:irMmmIw/7yzSRPWryHsK7EYSg09caPQL03VsM8rvUec=
github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs=
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
github.com/coreos/go-systemd/v22 v22.6.0 h1:aGVa/v8B7hpb0TKl0MWoAavPDmHvobFe5R5zn0bCJWo=
github.com/coreos/go-systemd/v22 v22.6.0/go.mod h1:iG+pp635Fo7ZmV/j14KUcmEyWF+0X7Lua8rrTWzYgWU=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 h1:HVTnpeuvF6Owjd5mniCL8DEXo7uYXdQEmOP4FJbV5tg=
github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3/go.mod h1:p1d6YEZWvFzEh4KLyvBcVSnrfNDDvK2zfK/4x2v/4pE=
@@ -231,8 +269,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/deckarep/golang-set/v2 v2.8.0 h1:swm0rlPCmdWn9mESxKOjWk8hXSqoxOp+ZlfuyaAdFlQ=
github.com/deckarep/golang-set/v2 v2.8.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4=
github.com/deckarep/golang-set/v2 v2.9.0 h1:prva4eP9UysWagLyKrtn074ughi0NnkIf0A4M5yOCKI=
github.com/deckarep/golang-set/v2 v2.9.0/go.mod h1:EWknQXbs0mcFpat2QOoXV0Ee57cD+w6ZEN76BR2JVrM=
github.com/decred/dcrd/crypto/blake256 v1.0.0 h1:/8DMNYp9SGi5f0w7uCm6d6M4OU2rGFK09Y2A4Xv7EE0=
github.com/decred/dcrd/crypto/blake256 v1.0.0/go.mod h1:sQl2p6Y26YV+ZOcSTP6thNdn47hh8kt6rqSlvmrXFAc=
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 h1:HbphB4TFFXpv7MNrT52FGrrgVXF1owhMVTHFZIlnvd4=
@@ -241,8 +279,8 @@ github.com/dhowden/tag v0.0.0-20240417053706-3d75831295e8 h1:OtSeLS5y0Uy01jaKK4m
github.com/dhowden/tag v0.0.0-20240417053706-3d75831295e8/go.mod h1:apkPC/CR3s48O2D7Y++n1XWEpgPNNCjXYga3PPbJe2E=
github.com/disintegration/imaging v1.6.2 h1:w1LecBlG2Lnp8B3jk5zSuNqd7b4DXhcjwek1ei82L+c=
github.com/disintegration/imaging v1.6.2/go.mod h1:44/5580QXChDfwIclfc/PCwrr44amcmDAg8hxG0Ewe4=
github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=
github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/dlclark/regexp2 v1.12.0 h1:0j4c5qQmnC6XOWNjP3PIXURXN2gWx76rd3KvgdPkCz8=
github.com/dlclark/regexp2 v1.12.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 h1:2tV76y6Q9BB+NEBasnqvs7e49aEBFI8ejC89PSnWH+4=
github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707/go.mod h1:qssHWj60/X5sZFNxpG4HBPDHVqxNm4DfnCKgrbZOT+s=
github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY=
@@ -250,8 +288,10 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/dustinxie/ecc v0.0.0-20210511000915-959544187564 h1:I6KUy4CI6hHjqnyJLNCEi7YHVMkwwtfSr2k9splgdSM=
github.com/dustinxie/ecc v0.0.0-20210511000915-959544187564/go.mod h1:yekO+3ZShy19S+bsmnERmznGy9Rfg6dWWWpiGJjNAz8=
github.com/ebitengine/purego v0.8.4 h1:CF7LEKg5FFOsASUj0+QwaXf8Ht6TlFxg09+S9wz0omw=
github.com/ebitengine/purego v0.8.4/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/ebitengine/purego v0.10.0 h1:QIw4xfpWT6GWTzaW5XEKy3HXoqrJGx1ijYHzTF0/ISU=
github.com/ebitengine/purego v0.10.0/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/ebitengine/purego v0.10.2 h1:W809HbnvzAxgdm+aOvlSekrM16wGCdT/e76+9tS7gzE=
github.com/ebitengine/purego v0.10.2/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
github.com/emersion/go-message v0.18.2 h1:rl55SQdjd9oJcIoQNhubD2Acs1E6IzlZISRTK7x/Lpg=
github.com/emersion/go-message v0.18.2/go.mod h1:XpJyL70LwRvq2a8rVbHXikPgKj8+aI0kGdHlg16ibYA=
github.com/emersion/go-vcard v0.0.0-20241024213814-c9703dde27ff h1:4N8wnS3f1hNHSmFD5zgFkWCyA4L1kCDkImPAtK7D6tg=
@@ -271,22 +311,26 @@ github.com/foxxorcat/weiyun-sdk-go v0.1.4/go.mod h1:TPxzN0d2PahweUEHlOBWlwZSA+rE
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fxamacker/cbor/v2 v2.9.0 h1:NpKPmjDBgUfBms6tr6JZkTHtfFGcMKsw3eGcmD/sapM=
github.com/fxamacker/cbor/v2 v2.9.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
github.com/gabriel-vasile/mimetype v1.4.9 h1:5k+WDwEsD9eTLL8Tz3L0VnmVh9QxGjRmjBvAG7U/oYY=
github.com/gabriel-vasile/mimetype v1.4.9/go.mod h1:WnSQhFKJuBlRyLiKohA/2DtIlPFAbguNaG7QCHcyGok=
github.com/gabriel-vasile/mimetype v1.4.12 h1:e9hWvmLYvtp846tLHam2o++qitpguFiYCKbn0w9jyqw=
github.com/gabriel-vasile/mimetype v1.4.12/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
github.com/gabriel-vasile/mimetype v1.4.13 h1:46nXokslUBsAJE/wMsp5gtO500a4F3Nkz9Ufpk2AcUM=
github.com/gabriel-vasile/mimetype v1.4.13/go.mod h1:d+9Oxyo1wTzWdyVUPMmXFvp4F9tea18J8ufA774AB3s=
github.com/geoffgarside/ber v1.2.0 h1:/loowoRcs/MWLYmGX9QtIAbA+V/FrnVLsMMPhwiRm64=
github.com/geoffgarside/ber v1.2.0/go.mod h1:jVPKeCbj6MvQZhwLYsGwaGI52oUorHoHKNecGT85ZCc=
github.com/gin-contrib/cors v1.7.6 h1:3gQ8GMzs1Ylpf70y8bMw4fVpycXIeX1ZemuSQIsnQQY=
github.com/gin-contrib/cors v1.7.6/go.mod h1:Ulcl+xN4jel9t1Ry8vqph23a60FwH9xVLd+3ykmTjOk=
github.com/gin-contrib/cors v1.7.7 h1:Oh9joP463x7Mw72vhvJ61YQm8ODh9b04YR7vsOErD0Q=
github.com/gin-contrib/cors v1.7.7/go.mod h1:K5tW0RkzJtWSiOdikXloy8VEZlgdVNpHNw8FpjUPNrE=
github.com/gin-contrib/sse v1.1.0 h1:n0w2GMuUpWDVp7qSpvze6fAu9iRxJY4Hmj6AmBOU05w=
github.com/gin-contrib/sse v1.1.0/go.mod h1:hxRZ5gVpWMT7Z0B0gSNYqqsSCNIJMjzvm6fqCz9vjwM=
github.com/gin-gonic/gin v1.10.1 h1:T0ujvqyCSqRopADpgPgiTT63DUQVSfojyME59Ei63pQ=
github.com/gin-gonic/gin v1.10.1/go.mod h1:4PMNQiOhvDRa013RKVbsiNwoyezlm2rm0uX/T7kzp5Y=
github.com/gin-gonic/gin v1.12.0 h1:b3YAbrZtnf8N//yjKeU2+MQsh2mY5htkZidOM7O0wG8=
github.com/gin-gonic/gin v1.12.0/go.mod h1:VxccKfsSllpKshkBWgVgRniFFAzFb9csfngsqANjnLc=
github.com/glebarez/go-sqlite v1.21.2 h1:3a6LFC4sKahUunAmynQKLZceZCOzUthkRkEAl9gAXWo=
github.com/glebarez/go-sqlite v1.21.2/go.mod h1:sfxdZyhQjTM2Wry3gVYWaW072Ri1WMdWJi0k6+3382k=
github.com/glebarez/sqlite v1.11.0 h1:wSG0irqzP6VurnMEpFGer5Li19RpIRi2qvQz++w0GMw=
github.com/glebarez/sqlite v1.11.0/go.mod h1:h8/o8j5wiAsqSPoWELDUdJXhjAhsVliSn7bWZjOhrgQ=
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
github.com/go-chi/chi/v5 v5.2.2/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/go-chi/chi/v5 v5.2.5 h1:Eg4myHZBjyvJmAFjFvWgrqDTXFyOzjj7YIm3L3mu6Ug=
github.com/go-chi/chi/v5 v5.2.5/go.mod h1:X7Gx4mteadT3eDOMTsXzmI4/rwUpOwBHLpAfupzFJP0=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 h1:JnrjqG5iR07/8k7NqrLNilRsl3s1EPRQEGvbPyOce68=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348/go.mod h1:Czxo/d1g948LtrALAZdL04TL/HnkopquAjxYUuI02bo=
github.com/go-kit/log v0.2.1 h1:MRVx0/zhvdseW+Gza6N9rVzU/IVzaeE1SFI4raAhmBU=
@@ -307,18 +351,22 @@ github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/o
github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY=
github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY=
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.26.0 h1:SP05Nqhjcvz81uJaRfEV0YBSSSGMc/iMaVtFbr3Sw2k=
github.com/go-playground/validator/v10 v10.26.0/go.mod h1:I5QpIEbmr8On7W0TktmJAumgzX4CA1XNl4ZmDuVHKKo=
github.com/go-resty/resty/v2 v2.16.5 h1:hBKqmWrr7uRc3euHVqmh1HTHcKn99Smr7o5spptdhTM=
github.com/go-resty/resty/v2 v2.16.5/go.mod h1:hkJtXbA2iKHzJheXYvQ8snQES5ZLGKMwQ07xAwp/fiA=
github.com/go-playground/validator/v10 v10.30.1 h1:f3zDSN/zOma+w6+1Wswgd9fLkdwy06ntQJp0BBvFG0w=
github.com/go-playground/validator/v10 v10.30.1/go.mod h1:oSuBIQzuJxL//3MelwSLD5hc2Tu889bF0Idm9Dg26cM=
github.com/go-resty/resty/v2 v2.17.2 h1:FQW5oHYcIlkCNrMD2lloGScxcHJ0gkjshV3qcQAyHQk=
github.com/go-resty/resty/v2 v2.17.2/go.mod h1:kCKZ3wWmwJaNc7S29BRtUhJwy7iqmn+2mLtQrOyQlVA=
github.com/go-sql-driver/mysql v1.7.0 h1:ueSltNNllEqE3qcWBTD0iQd3IpL/6U+mJxLkazJ7YPc=
github.com/go-sql-driver/mysql v1.7.0/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI=
github.com/go-sql-driver/mysql v1.8.1 h1:LedoTUt/eveggdHS9qUFC1EFSa8bU2+1pZjSRpvNJ1Y=
github.com/go-sql-driver/mysql v1.8.1/go.mod h1:wEBSXgmK//2ZFJyE+qWnIsVGmvmEKlqwuVSjsCm7DZg=
github.com/go-webauthn/webauthn v0.13.4 h1:q68qusWPcqHbg9STSxBLBHnsKaLxNO0RnVKaAqMuAuQ=
github.com/go-webauthn/webauthn v0.13.4/go.mod h1:MglN6OH9ECxvhDqoq1wMoF6P6JRYDiQpC9nc5OomQmI=
github.com/go-webauthn/x v0.1.23 h1:9lEO0s+g8iTyz5Vszlg/rXTGrx3CjcD0RZQ1GPZCaxI=
github.com/go-webauthn/x v0.1.23/go.mod h1:AJd3hI7NfEp/4fI6T4CHD753u91l510lglU7/NMN6+E=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/goccy/go-yaml v1.19.2 h1:PmFC1S6h8ljIz6gMRBopkjP1TVT7xuwrButHID66PoM=
github.com/goccy/go-yaml v1.19.2/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o=
github.com/golang-jwt/jwt/v4 v4.5.1/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
@@ -326,6 +374,8 @@ github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXe
github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-jwt/jwt/v5 v5.2.3 h1:kkGXqQOBSDDWRhWNXTFpqGSCMyh/PLnqUvMGJPDJDs0=
github.com/golang-jwt/jwt/v5 v5.2.3/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
@@ -391,18 +441,20 @@ github.com/ipfs/boxo v0.12.0 h1:AXHg/1ONZdRQHQLgG5JHsSC3XoE4DjCAMgK+asZvUcQ=
github.com/ipfs/boxo v0.12.0/go.mod h1:xAnfiU6PtxWCnRqu7dcXQ10bB5/kvI1kXRotuGqGBhg=
github.com/ipfs/go-cid v0.5.0 h1:goEKKhaGm0ul11IHA7I6p1GmKz8kEYniqFopaB5Otwg=
github.com/ipfs/go-cid v0.5.0/go.mod h1:0L7vmeNXpQpUS9vt+yEARkJ8rOg43DF3iPgn4GIN0mk=
github.com/ipfs/go-cid v0.6.2 h1:VuGwJd+KJTaMJ4S4d5EEf9SXc17YUblS5axCbocn9YE=
github.com/ipfs/go-cid v0.6.2/go.mod h1:Xhwg8NzHeK9xPCEZkCw4idzPiuNMpX3fARuI5Iwj1Lo=
github.com/ipfs/go-ipfs-api v0.7.0 h1:CMBNCUl0b45coC+lQCXEVpMhwoqjiaCwUIrM+coYW2Q=
github.com/ipfs/go-ipfs-api v0.7.0/go.mod h1:AIxsTNB0+ZhkqIfTZpdZ0VR/cpX5zrXjATa3prSay3g=
github.com/itsHenry35/gofakes3 v0.0.8 h1:1AgOl04IgoUV5r/WSK7ycnvwfpgharYLfVTmnzk5miw=
github.com/itsHenry35/gofakes3 v0.0.8/go.mod h1:gQwOJ7LoH5QSpCVmjzC6oKp+MS71utLS7GHtonsvD0c=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a h1:bbPeKD0xmW/Y25WS6cokEszi5g+S0QxI/d45PkRi7Nk=
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.5.5 h1:amBjrZVmksIdNjxGW/IiIMzxMKZFelXbUoPNb+8sjQw=
github.com/jackc/pgx/v5 v5.5.5/go.mod h1:ez9gk+OAat140fv9ErkZDYFWmXLfV+++K0uAOiwgm1A=
github.com/jackc/puddle/v2 v2.2.1 h1:RhxXJtFG022u4ibrCSMSiu5aOq1i77R3OHKNJj77OAk=
github.com/jackc/puddle/v2 v2.2.1/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.6.0 h1:SWJzexBzPL5jb0GEsrPMLIsi/3jOo7RHlzTjcAeDrPY=
github.com/jackc/pgx/v5 v5.6.0/go.mod h1:DNZ/vlrUnhWCoFGxHAG8U2ljioxukquj7utPDgtQdTw=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/jcmturner/aescts/v2 v2.0.0 h1:9YKLH6ey7H4eDBXW8khjYslgyqG2xZikXP0EQFKrle8=
github.com/jcmturner/aescts/v2 v2.0.0/go.mod h1:AiaICIRyfYg35RUkr8yESTqvSy7csK90qZ5xfvvsoNs=
github.com/jcmturner/dnsutils/v2 v2.0.0 h1:lltnkeZGL0wILNvrNiVCR6Ro5PGU/SeBvVO/8c/iPbo=
@@ -421,6 +473,8 @@ github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ=
github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8=
github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3 h1:ZxO6Qr2GOXPdcW80Mcn3nemvilMPvpWqxrNfK2ZnNNs=
github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3/go.mod h1:dvLUr/8Fs9a2OBrEnCC5duphbkz/k/mSy5OkXg3PAgI=
github.com/jlaffaye/ftp v0.2.1-0.20251026020404-6602e981a1bb h1:6vkM8gO+zFV2m21QzGYyUSq5TP0VQgP2Xz3UQyCN2kI=
github.com/jlaffaye/ftp v0.2.1-0.20251026020404-6602e981a1bb/go.mod h1:H1+whwD0Qe3YOunlXIWhh3rlvzW5cZfkMDYGQPg+KAM=
github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
@@ -437,15 +491,14 @@ github.com/kdomanski/iso9660 v0.4.0/go.mod h1:OxUSupHsO9ceI8lBLPJKWBTphLemjrCQY8
github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/klauspost/compress v1.19.0 h1:sXLILfc9jV2QYWkzFOPWStmcUVH2RHEB1JCdY2oVvCQ=
github.com/klauspost/compress v1.19.0/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/klauspost/pgzip v1.2.6 h1:8RXeL5crjEUFnR2/Sn6GJNWtSQ3Dk8pq4CL3jvdDyjU=
github.com/klauspost/pgzip v1.2.6/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
github.com/knz/go-libedit v1.10.1/go.mod h1:MZTVkCWyz0oBc7JOWP3wNAzd002ZbM/5hgShxwh4x8M=
github.com/kr/fs v0.1.0 h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
@@ -456,6 +509,8 @@ github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/lanrat/extsort v1.0.2 h1:p3MLVpQEPwEGPzeLBb+1eSErzRl6Bgjgr+qnIs2RxrU=
github.com/lanrat/extsort v1.0.2/go.mod h1:ivzsdLm8Tv+88qbdpMElV6Z15StlzPUtZSKsGb51hnQ=
github.com/lanrat/extsort v1.4.2 h1:akbLIdo4PhNZtvjpaWnbXtGMmLtnGzXplkzfgl+XTTY=
github.com/lanrat/extsort v1.4.2/go.mod h1:hceP6kxKPKebjN1RVrDBXMXXECbaI41Y94tt6MDazc4=
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
github.com/libp2p/go-buffer-pool v0.1.0 h1:oK4mSFcQz7cTQIfqbe4MIj9gLW+mnanjyFtc6cdF0Y8=
@@ -464,14 +519,18 @@ github.com/libp2p/go-flow-metrics v0.1.0 h1:0iPhMI8PskQwzh57jB9WxIuIOQ0r+15PChFG
github.com/libp2p/go-flow-metrics v0.1.0/go.mod h1:4Xi8MX8wj5aWNDAZttg6UPmc0ZrnFNsMtpsYUClFtro=
github.com/libp2p/go-libp2p v0.27.8 h1:IX5x/4yKwyPQeVS2AXHZ3J4YATM9oHBGH1gBc23jBAI=
github.com/libp2p/go-libp2p v0.27.8/go.mod h1:eCFFtd0s5i/EVKR7+5Ki8bM7qwkNW3TPTTSSW9sz8NE=
github.com/lucasb-eyer/go-colorful v1.2.0 h1:1nnpGOrhyZZuNyfu1QjKiUICQ74+3FNCN69Aj6K7nkY=
github.com/lucasb-eyer/go-colorful v1.2.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag=
github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35 h1:PpXWgLPs+Fqr325bN2FD2ISlRRztXibcX6e8f5FR5Dc=
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/lufia/plan9stats v0.0.0-20260216142805-b3301c5f2a88 h1:PTw+yKnXcOFCR6+8hHTyWBeQ/P4Nb7dd4/0ohEcWQuM=
github.com/lufia/plan9stats v0.0.0-20260216142805-b3301c5f2a88/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/mailru/easyjson v0.9.0 h1:PrnmzHw7262yW8sTBwxi1PdJA3Iw/EKBa8psRf7d9a4=
github.com/mailru/easyjson v0.9.0/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
github.com/maruel/natural v1.1.1 h1:Hja7XhhmvEFhcByqDoHz9QZbkWey+COd9xWfCfn1ioo=
github.com/maruel/natural v1.1.1/go.mod h1:v+Rfd79xlw1AgVBjbO0BEQmptqb5HvL/k9GRHB7ZKEg=
github.com/mailru/easyjson v0.9.1 h1:LbtsOm5WAswyWbvTEOqhypdPeZzHavpZx96/n553mR8=
github.com/mailru/easyjson v0.9.1/go.mod h1:1+xMtQp2MRNVL/V1bOzuP3aP8VNwRW55fQUto+XFtTU=
github.com/maruel/natural v1.3.0 h1:VsmCsBmEyrR46RomtgHs5hbKADGRVtliHTyCOLFBpsg=
github.com/maruel/natural v1.3.0/go.mod h1:v+Rfd79xlw1AgVBjbO0BEQmptqb5HvL/k9GRHB7ZKEg=
github.com/matoous/go-nanoid/v2 v2.1.0 h1:P64+dmq21hhWdtvZfEAofnvJULaRR1Yib0+PnU669bE=
github.com/matoous/go-nanoid/v2 v2.1.0/go.mod h1:KlbGNQ+FhrUNIHUxZdL63t7tl4LaPkZNpUULS8H4uVM=
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
@@ -480,20 +539,22 @@ github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWE
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-localereader v0.0.1 h1:ygSAOl7ZXTx4RdPYinUpg6W99U8jWvWi9Ye2JC/oIi4=
github.com/mattn/go-localereader v0.0.1/go.mod h1:8fBrzywKY7BI3czFoHkuzRoWE9C+EiG4R1k4Cjx5p88=
github.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc=
github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mattn/go-runewidth v0.0.22 h1:76lXsPn6FyHtTY+jt2fTTvsMUCZq1k0qwRsAMuxzKAk=
github.com/mattn/go-runewidth v0.0.22/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU=
github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/meilisearch/meilisearch-go v0.32.0 h1:cWcycpONSH3VLTZ5npUl1O5aXPkNM0vUx6bywnYqGbE=
github.com/meilisearch/meilisearch-go v0.32.0/go.mod h1:aNtyuwurDg/ggxQIcKqWH6G9g2ptc8GyY7PLY4zMn/g=
github.com/mholt/archives v0.1.3 h1:aEAaOtNra78G+TvV5ohmXrJOAzf++dIlYeDW3N9q458=
github.com/mholt/archives v0.1.3/go.mod h1:LUCGp++/IbV/I0Xq4SzcIR6uwgeh2yjnQWamjRQfLTU=
github.com/mholt/archives v0.1.5 h1:Fh2hl1j7VEhc6DZs2DLMgiBNChUux154a1G+2esNvzQ=
github.com/mholt/archives v0.1.5/go.mod h1:3TPMmBLPsgszL+1As5zECTuKwKvIfj6YcwWPpeTAXF4=
github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
github.com/microcosm-cc/bluemonday v1.0.27/go.mod h1:jFi9vgW+H7c3V0lb6nR74Ib/DIB5OBs92Dimizgw2cA=
github.com/mikelolasagasti/xz v1.0.1 h1:Q2F2jX0RYJUG3+WsM+FJknv+6eVjsjXNDV0KJXZzkD0=
github.com/mikelolasagasti/xz v1.0.1/go.mod h1:muAirjiOUxPRXwm9HdDtB3uoRPrGnL85XHtokL9Hcgc=
github.com/minio/minlz v1.0.0 h1:Kj7aJZ1//LlTP1DM8Jm7lNKvvJS2m74gyyXXn3+uJWQ=
github.com/minio/minlz v1.0.0/go.mod h1:qT0aEB35q79LLornSzeDH75LBf3aH1MV+jB5w9Wasec=
github.com/minio/minlz v1.0.1 h1:OUZUzXcib8diiX+JYxyRLIdomyZYzHct6EShOKtQY2A=
github.com/minio/minlz v1.0.1/go.mod h1:qT0aEB35q79LLornSzeDH75LBf3aH1MV+jB5w9Wasec=
github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM=
github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8=
github.com/minio/xxml v0.0.3 h1:ZIpPQpfyG5uZQnqqC0LZuWtPk/WT8G/qkxvO6jb7zMU=
@@ -511,6 +572,8 @@ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9G
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o=
github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
github.com/mr-tron/base58 v1.3.0 h1:K6Y13R2h+dku0wOqKtecgRnBUBPrZzLZy5aIj8lCcJI=
github.com/mr-tron/base58 v1.3.0/go.mod h1:2BuubE67DCSWwVfx37JWNG8emOC0sHEU4/HpcYgCLX8=
github.com/mschoch/smat v0.2.0 h1:8imxQsjDm8yFEAVBe7azKmKSgzSkZXDuKkSq9374khM=
github.com/mschoch/smat v0.2.0/go.mod h1:kc9mz7DoBKqDyiRL7VZN8KvXQMWeTaVnttLRXOlotKw=
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 h1:ZK8zHtRHOkbHy6Mmr5D264iyp3TiX5OmNcI5cIARiQI=
@@ -527,6 +590,8 @@ github.com/multiformats/go-multiaddr v0.9.0 h1:3h4V1LHIk5w4hJHekMKWALPXErDfz/sgg
github.com/multiformats/go-multiaddr v0.9.0/go.mod h1:mI67Lb1EeTOYb8GQfL/7wpIZwc46ElrvzhYnoJOmTT0=
github.com/multiformats/go-multibase v0.2.0 h1:isdYCVLvksgWlMW9OZRYJEa9pZETFivncJHmHnnd87g=
github.com/multiformats/go-multibase v0.2.0/go.mod h1:bFBZX4lKCA/2lyOFSAoKH5SS6oPyjtnzK/XTFDPkNuk=
github.com/multiformats/go-multibase v0.3.0 h1:8helZD2+4Db7NNWFiktk2NePbF0boolBe6bDQvM4r68=
github.com/multiformats/go-multibase v0.3.0/go.mod h1:MoBLQPCkRTOL3eveIPO81860j2AQY8JwcnNlRkGRUfI=
github.com/multiformats/go-multicodec v0.9.0 h1:pb/dlPnzee/Sxv/j4PmkDRxCOi3hXTz3IbPKOXWJkmg=
github.com/multiformats/go-multicodec v0.9.0/go.mod h1:L3QTQvMIaVBkXOXXtVmYE+LI16i14xuaojr/H7Ai54k=
github.com/multiformats/go-multihash v0.2.3 h1:7Lyc8XfX/IY2jWb/gI7JP+o7JEq9hOa7BFvVU9RSh+U=
@@ -535,26 +600,30 @@ github.com/multiformats/go-multistream v0.4.1 h1:rFy0Iiyn3YT0asivDUIR05leAdwZq3d
github.com/multiformats/go-multistream v0.4.1/go.mod h1:Mz5eykRVAjJWckE2U78c6xqdtyNUEhKSM0Lwar2p77Q=
github.com/multiformats/go-varint v0.0.7 h1:sWSGR+f/eu5ABZA2ZpYKBILXTTs9JWpdEM/nEGOHFS8=
github.com/multiformats/go-varint v0.0.7/go.mod h1:r8PUYw/fD/SjBCiKOoDlGF6QawOELpZAu9eioSos/OU=
github.com/multiformats/go-varint v0.1.0 h1:i2wqFp4sdl3IcIxfAonHQV9qU5OsZ4Ts9IOoETFs5dI=
github.com/multiformats/go-varint v0.1.0/go.mod h1:5KVAVXegtfmNQQm/lCY+ATvDzvJJhSkUlGQV9wgObdI=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/natefinch/lumberjack v2.0.0+incompatible h1:4QJd3OLAMgj7ph+yZTuX13Ld4UpgHp07nNdFX7mqFfM=
github.com/natefinch/lumberjack v2.0.0+incompatible/go.mod h1:Wi9p2TTF5DG5oU+6YfsmYQpsTIOm0B1VNzQg9Mw6nPk=
github.com/ncw/swift/v2 v2.0.4 h1:hHWVFxn5/YaTWAASmn4qyq2p6OyP/Hm3vMLzkjEqR7w=
github.com/ncw/swift/v2 v2.0.4/go.mod h1:cbAO76/ZwcFrFlHdXPjaqWZ9R7Hdar7HpjRXBfbjigk=
github.com/nwaples/rardecode/v2 v2.1.1 h1:OJaYalXdliBUXPmC8CZGQ7oZDxzX1/5mQmgn0/GASew=
github.com/nwaples/rardecode/v2 v2.1.1/go.mod h1:7uz379lSxPe6j9nvzxUZ+n7mnJNgjsRNb6IbvGVHRmw=
github.com/ncw/swift/v2 v2.0.5 h1:9o5Gsd7bInAFEqsGPcaUdsboMbqf8lnNtxqWKFT9iz8=
github.com/ncw/swift/v2 v2.0.5/go.mod h1:cbAO76/ZwcFrFlHdXPjaqWZ9R7Hdar7HpjRXBfbjigk=
github.com/nwaples/rardecode/v2 v2.2.5 h1:L5doqgGfQwI7qADJMqnkrSB86rpPsqQDrHeO0HWa5JY=
github.com/nwaples/rardecode/v2 v2.2.5/go.mod h1:7uz379lSxPe6j9nvzxUZ+n7mnJNgjsRNb6IbvGVHRmw=
github.com/nwaples/rardecode/v2 v2.3.0 h1:CtgyxWm8ClLcSh1u4M58fOz6lmeb/j4V7KpaEi/6UtM=
github.com/nwaples/rardecode/v2 v2.3.0/go.mod h1:7uz379lSxPe6j9nvzxUZ+n7mnJNgjsRNb6IbvGVHRmw=
github.com/panjf2000/ants/v2 v2.4.2/go.mod h1:f6F0NZVFsGCp5A7QW/Zj/m92atWwOkY0OIhFxRNFr4A=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pierrec/lz4/v4 v4.1.27 h1:+PhzhWDrjRj89TH2sw43nE3+4+W8lSxIuQadEHZyjUk=
github.com/pierrec/lz4/v4 v4.1.27/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/sftp v1.13.9 h1:4NGkvGudBL7GteO3m6qnaQ4pC0Kvf0onSVc9gR3EWBw=
github.com/pkg/sftp v1.13.9/go.mod h1:OBN7bVXdstkFFN/gdnHPUb5TE8eb8G1Rp9wCItqjkkA=
github.com/pkg/sftp v1.13.10 h1:+5FbKNTe5Z9aspU88DPIKJ9z2KZoaGCu6Sr6kKR/5mU=
github.com/pkg/sftp v1.13.10/go.mod h1:bJ1a7uDhrX/4OII+agvy28lzRvQrmIQuaHrcI1HbeGA=
github.com/pkg/xattr v0.4.10 h1:Qe0mtiNFHQZ296vRgUjRCoPHPqH7VdTOrZx3g0T+pGA=
github.com/pkg/xattr v0.4.10/go.mod h1:di8WF84zAKk8jzR1UBTEWh9AUlIZZ7M/JNt8e9B6ktU=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
@@ -568,49 +637,62 @@ github.com/pquerna/otp v1.5.0 h1:NMMR+WrmaqXU4EzdGJEE1aUUI0AMRzsp96fFFWNPwxs=
github.com/pquerna/otp v1.5.0/go.mod h1:dkJfzwRKNiegxyNb54X/3fLwhCynbMspSyWKnvi1AEg=
github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.64.0 h1:pdZeA+g617P7oGv1CzdTzyeShxAGrTBsolKNOLQPGO4=
github.com/prometheus/common v0.64.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
github.com/prometheus/common v0.67.5 h1:pIgK94WWlQt1WLwAC5j2ynLaBRDiinoAb86HZHTUGI4=
github.com/prometheus/common v0.67.5/go.mod h1:SjE/0MzDEEAyrdr5Gqc6G+sXI67maCxzaT3A2+HqjUw=
github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
github.com/quic-go/qpack v0.5.1 h1:giqksBPnT/HDtZ6VhtFKgoLOWmlyo9Ei6u9PqzIMbhI=
github.com/quic-go/qpack v0.5.1/go.mod h1:+PC4XFrEskIVkcLzpEkbLqq1uCoxPhQuvK5rH1ZgaEg=
github.com/quic-go/quic-go v0.54.1 h1:4ZAWm0AhCb6+hE+l5Q1NAL0iRn/ZrMwqHRGQiFwj2eg=
github.com/quic-go/quic-go v0.54.1/go.mod h1:e68ZEaCdyviluZmy44P6Iey98v/Wfz6HCjQEm+l8zTY=
github.com/prometheus/procfs v0.19.2 h1:zUMhqEW66Ex7OXIiDkll3tl9a1ZdilUOd/F6ZXw4Vws=
github.com/prometheus/procfs v0.19.2/go.mod h1:M0aotyiemPhBCM0z5w87kL22CxfcH05ZpYlu+b4J7mw=
github.com/quic-go/go-ossfuzz-seeds v0.1.0 h1:APacT+iIaNF6fd8AGEiN3bT/Jtkd2jz4v4TzM7MFjy0=
github.com/quic-go/go-ossfuzz-seeds v0.1.0/go.mod h1:3IOHRbJIc+L6YKMwfDtJAM9Vj9k0YY4muhuyUYk5tbk=
github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
github.com/quic-go/quic-go v0.60.0 h1:xcQioE8OM66UQLeUMHltK1CCcOu3JbVB4JAQdDQSB+0=
github.com/quic-go/quic-go v0.60.0/go.mod h1:wpKpjmPpftl30sL6pFh7REVpjbcCVy4zt2vDyK1TuJk=
github.com/rclone/rclone v1.70.3 h1:rg/WNh4DmSVZyKP2tHZ4lAaWEyMi7h/F0r7smOMA3IE=
github.com/rclone/rclone v1.70.3/go.mod h1:nLyN+hpxAsQn9Rgt5kM774lcRDad82x/KqQeBZ83cMo=
github.com/rclone/rclone v1.74.4 h1:/c6LMO2kPQjJa6a/PBFcIYJHDvfNOAtfDBJy44mGMpc=
github.com/rclone/rclone v1.74.4/go.mod h1:M8GXX+n9vrP5HwMt0O6wxQcAjlUTUdSQugFrjXQI8kI=
github.com/relvacode/iso8601 v1.6.0 h1:eFXUhMJN3Gz8Rcq82f9DTMW0svjtAVuIEULglM7QHTU=
github.com/relvacode/iso8601 v1.6.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I=
github.com/relvacode/iso8601 v1.7.0 h1:BXy+V60stMP6cpswc+a93Mq3e65PfXCgDFfhvNNGrdo=
github.com/relvacode/iso8601 v1.7.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I=
github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rfjakob/eme v1.1.2 h1:SxziR8msSOElPayZNFfQw4Tjx/Sbaeeh3eRvrHVMUs4=
github.com/rfjakob/eme v1.1.2/go.mod h1:cVvpasglm/G3ngEfcfT/Wt0GwhkuO32pf/poW6Nyk1k=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rfjakob/eme v1.2.0 h1:8dAHL+WVAw06+7DkRKnRiFp1JL3QjcJEZFqDnndUaSI=
github.com/rfjakob/eme v1.2.0/go.mod h1:cVvpasglm/G3ngEfcfT/Wt0GwhkuO32pf/poW6Nyk1k=
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd h1:CmH9+J6ZSsIjUK3dcGsnCnO41eRBOnY12zwkn5qVwgc=
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk=
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46 h1:GHRpF1pTW19a8tTFrMLUcfWwyC0pnifVo2ClaLq+hP8=
github.com/ryszard/goskiplist v0.0.0-20150312221310-2dfbae5fcf46/go.mod h1:uAQ5PCi+MFsC7HjREoAz1BU+Mq60+05gifQSsHSDG/8=
github.com/secsy/goftp v0.0.0-20200609142545-aa2de14babf4 h1:PT+ElG/UUFMfqy5HrxJxNzj3QBOf7dZwupeVC+mG1Lo=
github.com/secsy/goftp v0.0.0-20200609142545-aa2de14babf4/go.mod h1:MnkX001NG75g3p8bhFycnyIjeQoOjGL6CEIsdE/nKSY=
github.com/shabbyrobe/gocovmerge v0.0.0-20230507112040-c3350d9342df h1:S77Pf5fIGMa7oSwp8SQPp7Hb4ZiI38K3RNBKD2LLeEM=
github.com/shabbyrobe/gocovmerge v0.0.0-20230507112040-c3350d9342df/go.mod h1:dcuzJZ83w/SqN9k4eQqwKYMgmKWzg/KzJAURBhRL1tc=
github.com/shirou/gopsutil/v4 v4.25.5 h1:rtd9piuSMGeU8g1RMXjZs9y9luK5BwtnG7dZaQUJAsc=
github.com/shirou/gopsutil/v4 v4.25.5/go.mod h1:PfybzyydfZcN+JMMjkF6Zb8Mq1A/VcogFFg7hj50W9c=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/shirou/gopsutil/v4 v4.26.6 h1:Mzr/npDtQC/xpeEuQKHZt8Zo9CmPvhTj8nkR8w5TLDs=
github.com/shirou/gopsutil/v4 v4.26.6/go.mod h1:LZ6ewCSkBqUpvSOf+LsTGnRinC6iaNUNMGBtDkJBaLQ=
github.com/shirou/gopsutil/v4 v4.26.7 h1:IXzpHz/dkMRYAhKkOXr1HB6SuzWU3eoyyeWe7g3bNZc=
github.com/shirou/gopsutil/v4 v4.26.7/go.mod h1:5O9FjBiXoTDFatIWjZZosqj4pV0DRtLx598xGbBehzM=
github.com/sirupsen/logrus v1.9.4 h1:TsZE7l11zFCLZnZ+teH4Umoq5BhEIfIzfRDZ1Uzql2w=
github.com/sirupsen/logrus v1.9.4/go.mod h1:ftWc9WdOfJ0a92nsE2jF5u5ZwH8Bv2zdeOC42RjbV2g=
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e h1:MRM5ITcdelLK2j1vwZ3Je0FKVCfqOLp5zO6trqMLYs0=
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e/go.mod h1:XV66xRDqSt+GTGFMVlhk3ULuV0y9ZmzeVGR4mloJI3M=
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966 h1:JIAuq3EEf9cgbU6AtGPK4CTG3Zf6CKMNqf0MHTggAUA=
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966/go.mod h1:sUM3LWHvSMaG192sy56D9F7CNvL7jUJVXoqM1QKLnog=
github.com/sorairolake/lzip-go v0.3.5 h1:ms5Xri9o1JBIWvOFAorYtUNik6HI3HgBTkISiqu0Cwg=
github.com/sorairolake/lzip-go v0.3.5/go.mod h1:N0KYq5iWrMXI0ZEXKXaS9hCyOjZUQdBDEIbXfoUwbdk=
github.com/sorairolake/lzip-go v0.3.8 h1:j5Q2313INdTA80ureWYRhX+1K78mUXfMoPZCw/ivWik=
github.com/sorairolake/lzip-go v0.3.8/go.mod h1:JcBqGMV0frlxwrsE9sMWXDjqn3EeVf0/54YPsw66qkU=
github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
@@ -621,6 +703,8 @@ github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiT
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stangelandcl/ppmd v0.1.1 h1:c25QazhlWUn5nmR1QOzafKhQxBicAr7GGCKER2aJ8H8=
github.com/stangelandcl/ppmd v0.1.1/go.mod h1:Rrv7M+/2P5jYr/GMLhBl7Ug3uJ1bUiVzr5LbbaV6xgY=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
@@ -635,35 +719,37 @@ github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/t3rm1n4l/go-mega v0.0.0-20241213151442-a19cff0ec7b5 h1:Sa+sR8aaAMFwxhXWENEnE6ZpqhZ9d7u1RT2722Rw6hc=
github.com/t3rm1n4l/go-mega v0.0.0-20241213151442-a19cff0ec7b5/go.mod h1:UdZiFUFu6e2WjjtjxivwXWcwc1N/8zgbkBR9QNucUOY=
github.com/t3rm1n4l/go-mega v0.0.0-20260630094356-4de2f127f34a h1:VQ9CmpYeTgdFvg748N6tIOHBZn+hXjJe2xNuOJHW07E=
github.com/t3rm1n4l/go-mega v0.0.0-20260630094356-4de2f127f34a/go.mod h1:BF/l2jNyK+2h/BJZ7VLMAz6m/IWjA2F67gTjV1C/+Bo=
github.com/t3rm1n4l/go-mega v0.0.0-20260717075258-c6acd6a5bd04 h1:s30A8dMuZ55lUOi5xUTh1hlfLqHpVBsRMfpk2iKCayk=
github.com/t3rm1n4l/go-mega v0.0.0-20260717075258-c6acd6a5bd04/go.mod h1:BF/l2jNyK+2h/BJZ7VLMAz6m/IWjA2F67gTjV1C/+Bo=
github.com/taruti/bytepool v0.0.0-20160310082835-5e3a9ea56543 h1:6Y51mutOvRGRx6KqyMNo//xk8B8o6zW9/RVmy1VamOs=
github.com/taruti/bytepool v0.0.0-20160310082835-5e3a9ea56543/go.mod h1:jpwqYA8KUVEvSUJHkCXsnBRJCSKP1BMa81QZ6kvRpow=
github.com/tchap/go-patricia/v2 v2.3.3 h1:xfNEsODumaEcCcY3gI0hYPZ/PcpVv5ju6RMAhgwZDDc=
github.com/tchap/go-patricia/v2 v2.3.3/go.mod h1:VZRHKAb53DLaG+nA9EaYYiaEx6YztwDlLElMsnSHD4k=
github.com/tklauser/go-sysconf v0.3.15 h1:VE89k0criAymJ/Os65CSn1IXaol+1wrsFHEB8Ol49K4=
github.com/tklauser/go-sysconf v0.3.15/go.mod h1:Dmjwr6tYFIseJw7a3dRLJfsHAMXZ3nEnL/aZY+0IuI4=
github.com/tklauser/numcpus v0.10.0 h1:18njr6LDBk1zuna922MgdjQuJFjrdppsZG60sHGfjso=
github.com/tklauser/numcpus v0.10.0/go.mod h1:BiTKazU708GQTYF4mB+cmlpT2Is1gLk7XVuEeem8LsQ=
github.com/tklauser/go-sysconf v0.3.16 h1:frioLaCQSsF5Cy1jgRBrzr6t502KIIwQ0MArYICU0nA=
github.com/tklauser/go-sysconf v0.3.16/go.mod h1:/qNL9xxDhc7tx3HSRsLWNnuzbVfh3e7gh/BmM179nYI=
github.com/tklauser/numcpus v0.11.0 h1:nSTwhKH5e1dMNsCdVBukSZrURJRoHbSEQjdEbY+9RXw=
github.com/tklauser/numcpus v0.11.0/go.mod h1:z+LwcLq54uWZTX0u/bGobaV34u6V7KNlTZejzM6/3MQ=
github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
github.com/u2takey/ffmpeg-go v0.5.0 h1:r7d86XuL7uLWJ5mzSeQ03uvjfIhiJYvsRAJFCW4uklU=
github.com/u2takey/ffmpeg-go v0.5.0/go.mod h1:ruZWkvC1FEiUNjmROowOAps3ZcWxEiOpFoHCvk97kGc=
github.com/u2takey/go-utils v0.3.1 h1:TaQTgmEZZeDHQFYfd+AdUT1cT4QJgJn/XVPELhHw4ys=
github.com/u2takey/go-utils v0.3.1/go.mod h1:6e+v5vEZ/6gu12w/DC2ixZdZtCrNokVxD0JUklcqdCs=
github.com/ugorji/go/codec v1.3.0 h1:Qd2W2sQawAfG8XSvzwhBeoGq71zXOC/Q1E9y/wUcsUA=
github.com/ugorji/go/codec v1.3.0/go.mod h1:pRBVtBSKl77K30Bv8R2P+cLSGaTtex6fsA2Wjqmfxj4=
github.com/ugorji/go/codec v1.3.1 h1:waO7eEiFDwidsBN6agj1vJQ4AG7lh2yqXyOXqhgQuyY=
github.com/ugorji/go/codec v1.3.1/go.mod h1:pRBVtBSKl77K30Bv8R2P+cLSGaTtex6fsA2Wjqmfxj4=
github.com/ulikunitz/xz v0.5.8/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/ulikunitz/xz v0.5.12 h1:37Nm15o69RwBkXM0J6A5OlE67RZTfzUxTj8fB3dfcsc=
github.com/ulikunitz/xz v0.5.12/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/ulikunitz/xz v0.5.15 h1:9DNdB5s+SgV3bQ2ApL10xRc35ck0DuIX/isZvIk+ubY=
github.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/unknwon/goconfig v1.0.0 h1:rS7O+CmUdli1T+oDm7fYj1MwqNWtEJfNj+FqcUHML8U=
github.com/unknwon/goconfig v1.0.0/go.mod h1:qu2ZQ/wcC/if2u32263HTVC39PeOQRSmidQk3DuDFQ8=
github.com/upyun/go-sdk/v3 v3.0.4 h1:2DCJa/Yi7/3ZybT9UCPATSzvU3wpPPxhXinNlb1Hi8Q=
github.com/upyun/go-sdk/v3 v3.0.4/go.mod h1:P/SnuuwhrIgAVRd/ZpzDWqCsBAf/oHg7UggbAxyZa0E=
github.com/winfsp/cgofuse v1.6.0 h1:re3W+HTd0hj4fISPBqfsrwyvPFpzqhDu8doJ9nOPDB0=
github.com/winfsp/cgofuse v1.6.0/go.mod h1:uxjoF2jEYT3+x+vC2KJddEGdk/LU8pRowXmyVMHSV5I=
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
@@ -673,34 +759,48 @@ github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3i
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
github.com/zeebo/blake3 v0.2.4 h1:KYQPkhpRtcqh0ssGYcKLG1JYvddkEA8QwCM/yBqhaZI=
github.com/zeebo/blake3 v0.2.4/go.mod h1:7eeQ6d2iXWRGF6npfaxl2CU+xy2Fjo2gxeyZGCRUjcE=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
github.com/zzzhr1990/go-common-entity v0.0.0-20250202070650-1a200048f0d3 h1:PSRwrE5QBufPnOjdgIkRs5KBV1Avq3SY8oksj2Z+k3o=
github.com/zzzhr1990/go-common-entity v0.0.0-20250202070650-1a200048f0d3/go.mod h1:CKriYB8bkNgSbYUQF1khSpejKb5IsV6cR7MdaAR7Fc0=
go.etcd.io/bbolt v1.4.0 h1:TU77id3TnN/zKr7CO/uk+fBCwF2jGcMuw2B/FMAzYIk=
go.etcd.io/bbolt v1.4.0/go.mod h1:AsD+OCi/qPN1giOX1aiLAha3o1U8rAz65bvN4j0sRuk=
go.etcd.io/bbolt v1.4.3 h1:dEadXpI6G79deX5prL3QRNP6JB8UxVkqo4UPnHaNXJo=
go.etcd.io/bbolt v1.4.3/go.mod h1:tKQlpPaYCVFctUIgFKFnAlvbmB3tpy1vkTnDWohtc0E=
go.mongodb.org/mongo-driver v1.17.4 h1:jUorfmVzljjr0FLzYQsGP8cgN/qzzxlY9Vh0C9KFXVw=
go.mongodb.org/mongo-driver v1.17.4/go.mod h1:Hy04i7O2kC4RS06ZrhPRqj/u4DTYkFDAAccj+rVKqgQ=
go.mongodb.org/mongo-driver v1.17.9 h1:IexDdCuuNJ3BHrELgBlyaH9p60JXAvdzWR128q+U5tU=
go.mongodb.org/mongo-driver v1.17.9/go.mod h1:LlOhpH5NUEfhxcAwG0UEkMqwYcc4JU18gtCdGudk/tQ=
go.mongodb.org/mongo-driver/v2 v2.5.0 h1:yXUhImUjjAInNcpTcAlPHiT7bIXhshCTL3jVBkF3xaE=
go.mongodb.org/mongo-driver/v2 v2.5.0/go.mod h1:yOI9kBsufol30iFsl1slpdq1I0eHPzybRWdyYUs8K/0=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 h1:F7Jx+6hwnZ41NSFTO5q4LYDtJRXBf2PD0rNBkeB/lus=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0/go.mod h1:UHB22Z8QsdRDrnAtX4PntOl36ajSxcdUMt1sF7Y6E7Q=
go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8=
go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM=
go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA=
go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI=
go.opentelemetry.io/otel/sdk v1.38.0 h1:l48sr5YbNf2hpCUj/FoGhW9yDkl+Ma+LrVl8qaM5b+E=
go.opentelemetry.io/otel/sdk v1.38.0/go.mod h1:ghmNdGlVemJI3+ZB5iDEuk4bWA3GkTpW+DOoZMYBVVg=
go.opentelemetry.io/otel/sdk/metric v1.38.0 h1:aSH66iL0aZqo//xXzQLYozmWrXxyFkBJ6qT5wthqPoM=
go.opentelemetry.io/otel/sdk/metric v1.38.0/go.mod h1:dg9PBnW9XdQ1Hd6ZnRz689CbtrUp0wMMs9iPcgT9EZA=
go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE=
go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs=
go.opentelemetry.io/otel v1.43.0 h1:mYIM03dnh5zfN7HautFE4ieIig9amkNANT+xcVxAj9I=
go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0=
go.opentelemetry.io/otel/metric v1.43.0 h1:d7638QeInOnuwOONPp4JAOGfbCEpYb+K6DVWvdxGzgM=
go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY=
go.opentelemetry.io/otel/sdk v1.43.0 h1:pi5mE86i5rTeLXqoF/hhiBtUNcrAGHLKQdhg4h4V9Dg=
go.opentelemetry.io/otel/sdk v1.43.0/go.mod h1:P+IkVU3iWukmiit/Yf9AWvpyRDlUeBaRg6Y+C58QHzg=
go.opentelemetry.io/otel/sdk/metric v1.43.0 h1:S88dyqXjJkuBNLeMcVPRFXpRw2fuwdvfCGLEo89fDkw=
go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A=
go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A=
go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
go.uber.org/goleak v1.2.1 h1:NBol2c7O1ZokfZ0LEU9K6Whx/KnwvepVetCUhtKja4A=
go.uber.org/goleak v1.2.1/go.mod h1:qlT2yGI9QafXHhZZLxlSuNsMw3FFLxBr+tBRlmO1xH4=
go.uber.org/mock v0.5.0 h1:KAMbZvZPyBPWgD14IrIQ38QCyjwpvVVV6K/bHl1IwQU=
go.uber.org/mock v0.5.0/go.mod h1:ge71pBPLYDk7QIi1LupWxdAykm7KIEFchiOqd6z7qMM=
go.uber.org/mock v0.6.0 h1:hyF9dfmbgIX5EfOdasqLsWD6xqpNZlXblLB/Dbnwv3Y=
go.uber.org/mock v0.6.0/go.mod h1:KiVJ4BqZJaMj4svdfmHM0AUx4NJYO8ZNpPnZn1Z+BBU=
go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0=
go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go4.org v0.0.0-20260112195520-a5071408f32f h1:ziUVAjmTPwQMBmYR1tbdRFJPtTcQUI12fH9QQjfb0Sw=
go4.org v0.0.0-20260112195520-a5071408f32f/go.mod h1:ZRJnO5ZI4zAwMFp+dS1+V6J6MSyAowhRqAE+DPa1Xp0=
gocv.io/x/gocv v0.25.0/go.mod h1:Rar2PS6DV+T4FL+PM535EImD/h13hGVaHhnCu1xarBs=
golang.org/x/arch v0.18.0 h1:WN9poc33zL4AzGxqf8VtpKUnGvMi8O9lhNyBMF/85qc=
golang.org/x/arch v0.18.0/go.mod h1:bdwinDaKcfZUGpH09BB7ZmOfhalA8lQdzl62l8gGWsk=
golang.org/x/arch v0.23.0 h1:lKF64A2jF6Zd8L0knGltUnegD62JMFBiCPBmQpToHhg=
golang.org/x/arch v0.23.0/go.mod h1:dNHoOeKiyja7GTvF9NJS1l3Z2yntpQNzgrjh1cU103A=
golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
@@ -711,20 +811,24 @@ golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliY
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/exp v0.0.0-20250606033433-dcc06ee1d476 h1:bsqhLWFR6G6xiQcb+JoGqdKdRU6WzPWmK8E0jxTjzo4=
golang.org/x/exp v0.0.0-20250606033433-dcc06ee1d476/go.mod h1:3//PLf8L/X+8b4vuAfHzxeRUl04Adcb341+IGKfnqS8=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f h1:W3F4c+6OLc6H2lb//N1q4WpJkhzJCK5J6kUi1NTVXfM=
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f/go.mod h1:J1xhfL/vlindoeF/aINzNzt2Bket5bjo9sdOYzOsU80=
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.29.0 h1:HcdsyR4Gsuys/Axh0rDEmlBmB68rW1U9BUdB3UVHsas=
golang.org/x/image v0.29.0/go.mod h1:RVJROnf3SLK8d26OW91j4FrIHGbsJ8QnbEocVTOWQDA=
golang.org/x/image v0.43.0 h1:FLxcP4ec2350nTfOC8ysKtqYSIFbk/QGjw1ZHNP4tsY=
golang.org/x/image v0.43.0/go.mod h1:rrpelvGFt+kLPAjPM4HeWPgrl0FtafueU//e5N0qk/Q=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.30.0 h1:fDEXFVZ/fmCKProc/yAXXUijritrDzahmwwefnjoPFk=
golang.org/x/mod v0.30.0/go.mod h1:lAsf5O2EvJeSFMiBxXDki7sCgAxEUcZHXoXMKT4GJKc=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
@@ -739,10 +843,12 @@ golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
golang.org/x/oauth2 v0.34.0 h1:hqK/t4AKgbqWkdkcAeI8XLmbK+4m4G5YeQRrmiotGlw=
golang.org/x/oauth2 v0.34.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -750,8 +856,8 @@ golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -764,7 +870,6 @@ golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20211007075335-d3039528d8ac/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -774,8 +879,10 @@ golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.40.0 h1:DBZZqJ2Rkml6QMQsZywtnjnnGvHza6BTfYFWY9kjEWQ=
golang.org/x/sys v0.40.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
@@ -786,8 +893,10 @@ golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/term v0.38.0 h1:PQ5pkm/rLO6HnxFR7N2lJHOZX6Kez5Y1gDSJla6jo7Q=
golang.org/x/term v0.38.0/go.mod h1:bSEAKrOT1W+VSu9TSCMtoGEOUcKxOKgl3LE5QEF/xVg=
golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0=
golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
@@ -800,11 +909,13 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190829051458-42f498d34c4d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
@@ -813,20 +924,20 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/tools v0.39.0 h1:ik4ho21kwuQln40uelmciQPp9SipgNDdrafrYA4TmQQ=
golang.org/x/tools v0.39.0/go.mod h1:JnefbkDPyD8UU2kI5fuf8ZX4/yUeh9W877ZeBONxUqQ=
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/api v0.259.0 h1:90TaGVIxScrh1Vn/XI2426kRpBqHwWIzVBzJsVZ5XrQ=
google.golang.org/api v0.259.0/go.mod h1:LC2ISWGWbRoyQVpxGntWwLWN/vLNxxKBK9KuJRI8Te4=
google.golang.org/appengine v1.6.8 h1:IhEN5q69dyKagZPYMSdIjS2HqprW324FRQZJcGqPAsM=
google.golang.org/appengine v1.6.8/go.mod h1:1jJ3jBArFh5pcgW8gCtRJnepW8FzD1V44FJffLiz/Ds=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b h1:Mv8VFug0MP9e5vUxfBcE3vUkV6CImK3cMNMIDFjmzxU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20251222181119-0a764e51fe1b/go.mod h1:j9x/tPzZkyxcgEFkiKEEGxfvyumM01BEtsW8xzOahRQ=
google.golang.org/grpc v1.78.0 h1:K1XZG/yGDJnzMdd/uZHAkVqJE+xIDOcmdSFZkBUicNc=
google.golang.org/grpc v1.78.0/go.mod h1:I47qjTo4OKbMkjA/aOOwxDIiPSBofUtQUI5EfpWvW7U=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478 h1:RmoJA1ujG+/lRGNfUnOMfhCy5EipVMyvUE+KNbPbTlw=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
google.golang.org/grpc v1.82.0 h1:vguDnZUPjE26w09A63VoxZPnvPjB5Riyc0mkXPFmAIU=
google.golang.org/grpc v1.82.0/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
@@ -842,6 +953,7 @@ gopkg.in/ldap.v3 v3.1.0 h1:DIDWEjI7vQWREh0S8X5/NFPCZ3MCVd55LmXKPW4XLGE=
gopkg.in/ldap.v3 v3.1.0/go.mod h1:dQjCc0R0kfyFjIlWNMH1DORwUASZyDxo2Ry1B51dXaQ=
gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k=
gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
@@ -853,13 +965,15 @@ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gorm.io/driver/mysql v1.5.7 h1:MndhOPYOfEp2rHKgkZIhJ16eVUIRf2HmzgoPmh7FCWo=
gorm.io/driver/mysql v1.5.7/go.mod h1:sEtPWMiqiN1N1cMXoXmBbd8C6/l+TESwriotuRRpkDM=
gorm.io/driver/postgres v1.5.9 h1:DkegyItji119OlcaLjqN11kHoUgZ/j13E0jkJZgD6A8=
gorm.io/driver/postgres v1.5.9/go.mod h1:DX3GReXH+3FPWGrrgffdvCk3DQ1dwDPdmbenSkweRGI=
gorm.io/driver/mysql v1.6.0 h1:eNbLmNTpPpTOVZi8MMxCi2aaIm0ZpInbORNXDwyLGvg=
gorm.io/driver/mysql v1.6.0/go.mod h1:D/oCC2GWK3M/dqoLxnOlaNKmXz8WNTfcS9y5ovaSqKo=
gorm.io/driver/postgres v1.6.0 h1:2dxzU8xJ+ivvqTRph34QX+WrRaJlmfyPqXmoGVjMBa4=
gorm.io/driver/postgres v1.6.0/go.mod h1:vUw0mrGgrTK+uPHEhAdV4sfFELrByKVGnaVRkXDhtWo=
gorm.io/driver/sqlite v1.6.0 h1:WHRRrIiulaPiPFmDcod6prc4l2VGVWHz80KspNsxSfQ=
gorm.io/driver/sqlite v1.6.0/go.mod h1:AO9V1qIQddBESngQUKWL9yoH93HIeA1X6V633rBwyT8=
gorm.io/gorm v1.25.7/go.mod h1:hbnx/Oo0ChWMn1BIhpy1oYozzpM15i4YPuHDmfYtwg8=
gorm.io/gorm v1.30.0 h1:qbT5aPv1UH8gI99OsRlvDToLxW5zR7FzS9acZDOZcgs=
gorm.io/gorm v1.30.0/go.mod h1:8Z33v652h4//uMA76KjeDH8mJXPm1QNCYrMeatR0DOE=
gorm.io/gorm v1.31.2 h1:3o8FXNo9v9S858gil+3LlZA1LkCOzgb4g5BL64FgaCo=
gorm.io/gorm v1.31.2/go.mod h1:XyQVbO2k6YkOis7C2437jSit3SsDK72s7n7rsSHd+Gs=
lukechampine.com/blake3 v1.1.7 h1:GgRMhmdsuK8+ii6UZFDL8Nb+VyMwadAgcJyfYHxG6n0=
lukechampine.com/blake3 v1.1.7/go.mod h1:tkKEOtDkNtklkXtLNEOGNq5tcV90tJiA1vAA12R78LA=
modernc.org/libc v1.22.5 h1:91BNch/e5B0uPbJFgqbxXuOnxBQjlS//icfQEGmvyjE=
@@ -870,7 +984,6 @@ modernc.org/memory v1.5.0 h1:N+/8c5rE6EqugZwHii4IFsaJ7MUhoWX07J5tC/iI5Ds=
modernc.org/memory v1.5.0/go.mod h1:PkUhL0Mugw21sHPeskwZW4D6VscE/GQJOnIpCnW6pSU=
modernc.org/sqlite v1.23.1 h1:nrSBg4aRQQwq59JpvGEQ15tNxoO5pX/kUjcRNwSAGQM=
modernc.org/sqlite v1.23.1/go.mod h1:OrDj17Mggn6MhE+iPbBNf7RGKODDE9NFT0f3EwDzJqk=
nullprogram.com/x/optparse v1.0.0/go.mod h1:KdyPE+Igbe0jQUrVfMqDMeJQIJZEuyV7pjYmp6pbG50=
resty.dev/v3 v3.0.0-beta.2 h1:xu4mGAdbCLuc3kbk7eddWfWm4JfhwDtdapwss5nCjnQ=
resty.dev/v3 v3.0.0-beta.2/go.mod h1:OgkqiPvTDtOuV4MGZuUDhwOpkY8enjOsjjMzeOHefy4=
sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc=
+2 -73
View File
@@ -249,79 +249,8 @@ func InitialSettings() []model.SettingItem {
{Key: conf.StreamMaxClientUploadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE},
{Key: conf.StreamMaxServerDownloadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE},
{Key: conf.StreamMaxServerUploadSpeed, Value: "-1", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE},
// media settings
{Key: conf.MediaTMDBKey, Value: "", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
{Key: conf.MediaTMDBAPIURL, Value: "api.themoviedb.org", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
{Key: conf.MediaDiscogsToken, Value: "", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
{Key: conf.MediaDiscogsAPIURL, Value: "api.discogs.com", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
{Key: conf.MediaStoreThumbnail, Value: "false", Type: conf.TypeBool, Group: model.MEDIA, Flag: model.PRIVATE},
{Key: conf.MediaThumbnailMode, Value: "base64", Type: conf.TypeSelect, Options: "base64,local", Group: model.MEDIA, Flag: model.PRIVATE},
{Key: conf.MediaThumbnailPath, Value: "/imgs", Type: conf.TypeString, Group: model.MEDIA, Flag: model.PRIVATE},
{Key: conf.MediaScrapeConcurrency, Value: "5", Type: conf.TypeNumber, Group: model.MEDIA, Flag: model.PRIVATE},
// transcode settings (FFmpeg 云端/本地转码) - 默认全部关闭
{Key: conf.TranscodeEnabled, Value: "false", Type: conf.TypeBool, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `开启后,超过阈值的媒体文件将通过 FFmpeg 转码后再播放;默认关闭`},
{Key: conf.TranscodeRunMode, Value: "local", Type: conf.TypeSelect, Options: "local,remote,hybrid", Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `local=仅使用本机内置 worker;remote=只使用远程 Worker 节点;hybrid=本地优先,超载后派发到远程`},
{Key: conf.TranscodeMinSizeGB, Value: "5", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `文件大于该 GB 数才走转码(小于则直链播放),0=任意大小都转码`},
{Key: conf.TranscodeMinBitrateMbps, Value: "20", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `视频码率超过该 Mbps 才转码,0=不限制`},
{Key: conf.TranscodeSourceCodecs, Value: "hevc,h265,av1,vvc,vp9", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `仅对这些源视频编码进行转码(逗号分隔)。常见高码率/兼容性差的编码:hevc,av1,vvc,vp9`},
{Key: conf.TranscodeSourceExtensions, Value: "mkv,ts,m2ts,mov,avi,wmv,flv,rmvb,webm", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `仅对这些后缀进行转码(逗号分隔,不带点),mp4 默认不转码可直接播放`},
{Key: conf.TranscodeOutputFormat, Value: "hls", Type: conf.TypeSelect, Options: "hls,dash,mp4", Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `输出封装格式,HLS 兼容性最好`},
{Key: conf.TranscodeOutputCodec, Value: "h264", Type: conf.TypeSelect, Options: "h264,hevc,av1", Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `重新编码后的视频编码,推荐 h264(最广兼容、互联网分发免授权费)`},
{Key: conf.TranscodeOutputBitrate, Value: "4000k", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `输出视频码率,例如 4000k / 6M。建议 1080p:4000k、720p:2500k`},
{Key: conf.TranscodeOutputAudioCodec, Value: "aac", Type: conf.TypeSelect, Options: "aac,mp3,opus,copy", Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `输出音频编码,aac 兼容性最好;copy=直接复制源音频流`},
{Key: conf.TranscodeOutputAudioBitrate, Value: "160k", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `输出音频码率`},
{Key: conf.TranscodeOutputResolution, Value: "1920x1080", Type: conf.TypeSelect, Options: "source,3840x2160,2560x1440,1920x1080,1280x720,854x480", Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `输出分辨率上限,超过该分辨率会下采样;source=保持源分辨率`},
{Key: conf.TranscodeSegmentDuration, Value: "6", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `HLS/DASH 切片时长(秒),越小首帧越快但请求数变多,推荐 4-10`},
{Key: conf.TranscodeHWAccel, Value: "none", Type: conf.TypeSelect,
Options: "none,auto,nvenc,qsv,vaapi,amf,videotoolbox",
Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `GPU 硬件加速:
none = 纯 CPU(libx264)
auto = 自动探测可用加速器
nvenc = NVIDIA GPU(GeForce/Tesla/Quadro/RTX,需 NVIDIA 驱动)
qsv = Intel 集显/独显 QuickSync(免费,功耗低)
vaapi = Linux 通用 VA-API(支持 Intel/AMD)
amf = AMD GPU(Windows AMF)
videotoolbox = macOS 硬件加速`},
{Key: conf.TranscodeFFmpegPath, Value: "ffmpeg", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `FFmpeg 可执行路径,留空则使用 PATH 中的 ffmpeg`},
{Key: conf.TranscodeFFprobePath, Value: "ffprobe", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `FFprobe 可执行路径`},
{Key: conf.TranscodeWorkerSecret, Value: "", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `远程 Worker 注册时使用的共享密钥,留空则禁用远程 Worker 注册`},
{Key: conf.TranscodeCachePath, Value: "data/transcode_cache", Type: conf.TypeString, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `转码切片缓存目录`},
{Key: conf.TranscodeCacheMaxGB, Value: "20", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `切片缓存最大容量(GB),超过后按 LRU 清理`},
{Key: conf.TranscodeJobTimeoutMin, Value: "120", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `单个转码任务超时分钟数,超时自动失败`},
{Key: conf.TranscodeLocalConcurrency, Value: "1", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `本机内置 worker 同时执行的转码任务数(local/hybrid 模式生效)`},
{Key: conf.TranscodeIdleTimeoutSec, Value: "90", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `播放端无请求多少秒后自动停止转码(默认90秒,0=禁用)`},
{Key: conf.TranscodeChunkDurationSec, Value: "60", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `每个 chunk 时长(秒),越大每段缓冲越长但 ffmpeg 进程切换越少(默认60秒)`},
{Key: conf.TranscodeMaxChunkConcurrency, Value: "2", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `单个视频任务最多同时运行的 chunk 数(默认2,越大并发越高但 CPU/GPU 占用越多)`},
{Key: conf.TranscodeChunkIdleSec, Value: "60", Type: conf.TypeNumber, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `chunk 多久没被访问就 kill ffmpeg 释放资源(默认60秒)`},
{Key: conf.TranscodeChunkPrefetch, Value: "true", Type: conf.TypeBool, Group: model.TRANSCODE, Flag: model.PRIVATE,
Help: `当前 chunk 转码进度过半时,是否预启动下一个 chunk(默认开启,可让顺序播放无缝衔接)`},
{Key: conf.MultipartEnabled, Value: "true", Type: conf.TypeBool, Group: model.TRAFFIC, Flag: model.PUBLIC},
{Key: conf.MultipartChunkSize, Value: "10", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PUBLIC, Help: `chunk size of multipart upload in MB (positive integer), keep it under your CDN's request body limit; each active session buffers up to 8 chunks on the server's disk`},
}
additionalSettingItems := tool.Tools.Items()
// 固定顺序
+2 -2
View File
@@ -20,12 +20,12 @@ import (
func InitDB() {
logLevel := logger.Silent
if flags.Debug || flags.Dev {
logLevel = logger.Warn // Warn 级别:只输出慢查询和错误,不输出每条 SQL
logLevel = logger.Info
}
newLogger := logger.New(
stdlog.New(log.StandardLogger().Out, "\r\n", stdlog.LstdFlags),
logger.Config{
SlowThreshold: 200 * time.Millisecond, // 超过 200ms 才记录慢查询
SlowThreshold: time.Second,
LogLevel: logLevel,
IgnoreRecordNotFoundError: true,
Colorful: true,
-5
View File
@@ -15,7 +15,6 @@ import (
"github.com/OpenListTeam/OpenList/v4/internal/conf"
"github.com/OpenListTeam/OpenList/v4/internal/db"
"github.com/OpenListTeam/OpenList/v4/internal/fs"
"github.com/OpenListTeam/OpenList/v4/internal/transcode"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/OpenListTeam/OpenList/v4/server"
"github.com/OpenListTeam/OpenList/v4/server/middlewares"
@@ -93,10 +92,6 @@ func Start() {
InitOfflineDownloadTools()
LoadStorages()
InitTaskManager()
// 初始化转码模块(总开关关时也会启动 manager,但本地 worker 仅在 enabled 时由 Manager.Start 内部按 run_mode 启动)
if transcode.IsEnabled() {
transcode.Default().Start()
}
if !flags.Debug && !flags.Dev {
gin.SetMode(gin.ReleaseMode)
}
+5 -39
View File
@@ -95,6 +95,9 @@ const (
// thunder_browser
ThunderBrowserTempDir = "thunder_browser_temp_dir"
// guangyapan
GuangYaPanTempDir = "guangyapan_temp_dir"
// single
Token = "token"
IndexProgress = "index_progress"
@@ -161,45 +164,8 @@ const (
StreamMaxClientUploadSpeed = "max_client_upload_speed"
StreamMaxServerDownloadSpeed = "max_server_download_speed"
StreamMaxServerUploadSpeed = "max_server_upload_speed"
// media
MediaTMDBKey = "media_tmdb_key"
MediaTMDBAPIURL = "media_tmdb_api_url"
MediaDiscogsToken = "media_discogs_token"
MediaDiscogsAPIURL = "media_discogs_api_url"
MediaThumbnailMode = "media_thumbnail_mode"
MediaThumbnailPath = "media_thumbnail_path"
MediaStoreThumbnail = "media_store_thumbnail"
MediaScrapeConcurrency = "media_scrape_concurrency"
// transcode (云端/本地 FFmpeg 转码)
TranscodeEnabled = "transcode_enabled" // 总开关,默认关
TranscodeMinSizeGB = "transcode_min_size_gb" // 文件大于多少 GB 才转码
TranscodeMinBitrateMbps = "transcode_min_bitrate_mbps" // 视频码率高于多少 Mbps 才转码(0=不限)
TranscodeSourceCodecs = "transcode_source_codecs" // 仅对这些源编码进行转码
TranscodeSourceExtensions = "transcode_source_extensions" // 仅对这些后缀进行转码
TranscodeOutputFormat = "transcode_output_format" // 输出封装格式 hls/dash/mp4
TranscodeOutputCodec = "transcode_output_codec" // 重新编码的视频编码
TranscodeOutputBitrate = "transcode_output_bitrate" // 输出视频码率,例如 4000k
TranscodeOutputAudioCodec = "transcode_output_audio_codec" // 输出音频编码
TranscodeOutputAudioBitrate = "transcode_output_audio_bitrate" // 输出音频码率,例如 160k
TranscodeOutputResolution = "transcode_output_resolution" // 输出分辨率上限
TranscodeSegmentDuration = "transcode_segment_duration" // HLS 切片时长(秒)
TranscodeHWAccel = "transcode_hwaccel" // 硬件加速类型 none/auto/nvenc/qsv/vaapi/amf/videotoolbox
TranscodeFFmpegPath = "transcode_ffmpeg_path" // 自定义 ffmpeg 可执行路径
TranscodeFFprobePath = "transcode_ffprobe_path" // 自定义 ffprobe 可执行路径
TranscodeRunMode = "transcode_run_mode" // local / remote / hybrid
TranscodeWorkerSecret = "transcode_worker_secret" // 远程 Worker 共享密钥
TranscodeCachePath = "transcode_cache_path" // 切片缓存目录
TranscodeCacheMaxGB = "transcode_cache_max_gb" // 缓存最大占用,超出 LRU 淘汰
TranscodeJobTimeoutMin = "transcode_job_timeout_min" // 单个任务超时分钟
TranscodeLocalConcurrency = "transcode_local_concurrency" // 本地内置 worker 并发数(run_mode=local/hybrid 时生效)
TranscodeIdleTimeoutSec = "transcode_idle_timeout_sec" // 播放端无请求多少秒后自动停止转码(默认90秒)
// === 智能 chunk 调度相关 ===
TranscodeChunkDurationSec = "transcode_chunk_duration_sec" // 每个 chunk 时长(秒),将整个视频分成多个 chunk 按需转码(默认60秒)
TranscodeMaxChunkConcurrency = "transcode_max_chunk_concurrency" // 单个 Job 最多同时运行的 chunk 数(默认2)
TranscodeChunkIdleSec = "transcode_chunk_idle_sec" // chunk 多久没被访问就 kill ffmpeg 释放资源(默认60秒)
TranscodeChunkPrefetch = "transcode_chunk_prefetch" // 当前 chunk 转码进度过半时,是否预启动下一个 chunk(默认true)
MultipartEnabled = "multipart_enabled"
MultipartChunkSize = "multipart_chunk_size"
)
const (
+1 -39
View File
@@ -12,50 +12,12 @@ var db *gorm.DB
func Init(d *gorm.DB) {
db = d
// 迁移前处理:处理 media_items 的存储语义变更,避免旧数据与新代码不一致
// 语义变更:folder_path 现为文件实际父目录(之前为扫描根路径),旧数据需重扫
migrateMediaItems()
err := AutoMigrate(new(model.Storage), new(model.User), new(model.Meta), new(model.SettingItem), new(model.SearchNode), new(model.TaskItem), new(model.SSHPublicKey), new(model.SharingDB), new(model.MediaItem), new(model.MediaConfig), new(model.MediaScanPath))
err := AutoMigrate(new(model.Storage), new(model.User), new(model.Meta), new(model.SettingItem), new(model.SearchNode), new(model.TaskItem), new(model.SSHPublicKey), new(model.SharingDB))
if err != nil {
log.Fatalf("failed migrate database: %s", err.Error())
}
}
// migrateMediaItems 处理 media_items 表的迁移兼容性
// 存储语义历史变更:
// v1: folder_path 为文件完整路径
// v2: folder_path 恒定为扫描根路径
// v3(当前): folder_path 为文件/文件夹的实际父目录,供前端拼接播放地址
// 唯一性由 folder_path + file_name + album_name 组合索引保证
func migrateMediaItems() {
// 检查表是否存在
if !db.Migrator().HasTable("x_media_items") {
return
}
// 语义标记:在设置表中写一个版本号,用来检测是否需要重扫。
// 这里采用轻量方式:检查是否存在以 sp.Path 开头的 folder_path 且可能丢了子路径的记录。
// 为避免复杂判断,直接在检测到旧版本索引时清空数据。
if db.Migrator().HasIndex("x_media_items", "idx_media_folder_file_album") {
// 已是新索引:仅清理软删除残留数据,避免与唯一索引冲突
if err := db.Exec("DELETE FROM x_media_items WHERE deleted_at IS NOT NULL").Error; err != nil {
log.Warnf("media_items: 清理软删除残留数据失败: %v", err)
}
return
}
// 旧表存在但没有新组合索引,说明是旧版本数据,需要清空后重建
// 旧数据的 folder_path 语义已变更,无法复用
log.Info("media_items: 检测到旧版本数据,清空后重新迁移(存储结构已变更)")
// 先尝试删除旧的单字段唯一索引(如果存在),避免 AutoMigrate 冲突
if db.Migrator().HasIndex("x_media_items", "idx_x_media_items_folder_path") {
if err := db.Migrator().DropIndex("x_media_items", "idx_x_media_items_folder_path"); err != nil {
log.Warnf("media_items: 删除旧唯一索引失败: %v", err)
}
}
if err := db.Exec("DELETE FROM x_media_items").Error; err != nil {
log.Warnf("media_items: 清空旧数据失败: %v", err)
}
}
func AutoMigrate(dst ...interface{}) error {
var err error
if conf.Conf.Database.Type == "mysql" {
-563
View File
@@ -1,563 +0,0 @@
package db
import (
"encoding/json"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"gorm.io/gorm"
)
// ==================== MediaConfig ====================
// GetMediaConfig 获取指定类型的媒体库配置,不存在则返回默认值
func GetMediaConfig(mediaType model.MediaType) (*model.MediaConfig, error) {
var cfg model.MediaConfig
result := db.Where("media_type = ?", mediaType).First(&cfg)
if result.Error == gorm.ErrRecordNotFound {
return &model.MediaConfig{
MediaType: mediaType,
Enabled: false,
}, nil
}
return &cfg, result.Error
}
// GetAllMediaConfigs 获取所有媒体库配置
func GetAllMediaConfigs() ([]model.MediaConfig, error) {
var cfgs []model.MediaConfig
err := db.Find(&cfgs).Error
return cfgs, err
}
// SaveMediaConfig 保存媒体库配置(upsert)
func SaveMediaConfig(cfg *model.MediaConfig) error {
var existing model.MediaConfig
result := db.Where("media_type = ?", cfg.MediaType).First(&existing)
if result.Error == gorm.ErrRecordNotFound {
return db.Create(cfg).Error
}
cfg.ID = existing.ID
return db.Save(cfg).Error
}
// ==================== MediaScanPath ====================
// ListMediaScanPaths 获取指定媒体类型的所有扫描路径
func ListMediaScanPaths(mediaType model.MediaType) ([]model.MediaScanPath, error) {
var paths []model.MediaScanPath
tx := db.Model(&model.MediaScanPath{})
if mediaType != "" {
tx = tx.Where("media_type = ?", mediaType)
}
err := tx.Order("id asc").Find(&paths).Error
return paths, err
}
// GetMediaScanPath 按ID获取扫描路径
func GetMediaScanPath(id uint) (*model.MediaScanPath, error) {
var p model.MediaScanPath
err := db.First(&p, id).Error
return &p, err
}
// CreateMediaScanPath 创建扫描路径
func CreateMediaScanPath(p *model.MediaScanPath) error {
return db.Create(p).Error
}
// UpdateMediaScanPath 更新扫描路径
func UpdateMediaScanPath(p *model.MediaScanPath) error {
return db.Save(p).Error
}
// DeleteMediaScanPath 删除扫描路径(硬删除)
func DeleteMediaScanPath(id uint) error {
return db.Unscoped().Delete(&model.MediaScanPath{}, id).Error
}
// ==================== MediaItem ====================
// MediaItemQuery 媒体条目查询参数
type MediaItemQuery struct {
MediaType model.MediaType
ScanPathID uint // 按扫描路径ID筛选
FolderPath string // 按文件夹路径筛选
TypeTag string // 按类型标签筛选(电影、电视剧等)
ContentTag string // 按内容标签筛选(喜剧、惊悚等)
Hidden *bool
Keyword string
OrderBy string // "name", "date", "size"
OrderDir string // "asc", "desc"
Page int
PageSize int
}
// ListMediaItems 分页查询媒体条目
func ListMediaItems(q MediaItemQuery) ([]model.MediaItem, int64, error) {
tx := db.Model(&model.MediaItem{})
if q.MediaType != "" {
tx = tx.Where("media_type = ?", q.MediaType)
}
if q.ScanPathID > 0 {
tx = tx.Where("scan_path_id = ?", q.ScanPathID)
}
if q.FolderPath != "" {
tx = tx.Where("folder_path = ?", q.FolderPath)
}
if q.Hidden != nil {
tx = tx.Where("hidden = ?", *q.Hidden)
}
if q.Keyword != "" {
like := "%" + q.Keyword + "%"
tx = tx.Where("file_name LIKE ? OR scraped_name LIKE ?", like, like)
}
// 按类型标签筛选(通过关联扫描路径的type_tag)
if q.TypeTag != "" {
tx = tx.Joins("JOIN media_scan_paths ON media_scan_paths.id = media_items.scan_path_id").
Where("media_scan_paths.type_tag = ?", q.TypeTag)
}
// 按内容标签筛选(通过关联扫描路径的content_tags)
if q.ContentTag != "" {
tx = tx.Joins("JOIN media_scan_paths ON media_scan_paths.id = media_items.scan_path_id").
Where("media_scan_paths.content_tags LIKE ?", "%"+q.ContentTag+"%")
}
var total int64
if err := tx.Count(&total).Error; err != nil {
return nil, 0, err
}
// 排序
orderCol := "created_at"
switch q.OrderBy {
case "name":
orderCol = "COALESCE(NULLIF(scraped_name,''), file_name)"
case "date":
orderCol = "release_date"
case "size":
orderCol = "file_size"
}
dir := "asc"
if q.OrderDir == "desc" {
dir = "desc"
}
tx = tx.Order(orderCol + " " + dir)
// 分页
if q.PageSize <= 0 {
q.PageSize = 20
}
if q.Page <= 0 {
q.Page = 1
}
tx = tx.Offset((q.Page - 1) * q.PageSize).Limit(q.PageSize)
var items []model.MediaItem
err := tx.Find(&items).Error
return items, total, err
}
// GetMediaItemByID 按ID获取媒体条目
func GetMediaItemByID(id uint) (*model.MediaItem, error) {
var item model.MediaItem
err := db.First(&item, id).Error
return &item, err
}
// GetMediaItemByFolderPath 按文件夹路径获取媒体条目(用于合并模式)
func GetMediaItemByFolderPath(folderPath string) (*model.MediaItem, error) {
var item model.MediaItem
result := db.Where("folder_path = ?", folderPath).First(&item)
return &item, result.Error
}
// CreateOrUpdateMediaItem 创建或更新媒体条目(按 folder_path + file_name + album_name 组合唯一)
// 更新时保留已有的刮削数据,避免重新扫描时把已刮削的字段清空
//
// 注意:使用 Unscoped() 查询是为了把软删除残留记录也包含进来,
// 否则唯一索引 idx_media_folder_file_album 会与软删除行冲突,
// 导致新建条目时报 UNIQUE constraint failed。
func CreateOrUpdateMediaItem(item *model.MediaItem) error {
var existing model.MediaItem
result := db.Unscoped().Where("folder_path = ? AND file_name = ? AND album_name = ?", item.FolderPath, item.FileName, item.AlbumName).First(&existing)
if result.Error == gorm.ErrRecordNotFound {
return db.Create(item).Error
}
if result.Error != nil {
return result.Error
}
item.ID = existing.ID
item.CreatedAt = existing.CreatedAt
// 复用已有记录时清除软删除标记,确保该条目“恢复”为正常记录
item.DeletedAt = gorm.DeletedAt{}
// 如果已有刮削数据,保留刮削字段,防止重新扫描时覆盖刮削结果
if existing.ScrapedAt != nil {
item.ScrapedAt = existing.ScrapedAt
item.ScrapedName = existing.ScrapedName
item.Cover = existing.Cover
item.AlbumName = existing.AlbumName
item.AlbumArtist = existing.AlbumArtist
item.TrackNumber = existing.TrackNumber
item.Duration = existing.Duration
item.Genre = existing.Genre
item.ReleaseDate = existing.ReleaseDate
item.Rating = existing.Rating
item.Plot = existing.Plot
item.Authors = existing.Authors
item.Description = existing.Description
item.Publisher = existing.Publisher
item.ISBN = existing.ISBN
item.ExternalID = existing.ExternalID
}
return db.Unscoped().Save(item).Error
}
// UpdateMediaItem 更新媒体条目
// 使用 Unscoped 确保即使存在软删除标记也能正确更新(避免命中唯一索引但行不可见的诡异情况)。
func UpdateMediaItem(item *model.MediaItem) error {
return db.Unscoped().Save(item).Error
}
// DeleteMediaItem 硬删除媒体条目(真正从数据库删除)
func DeleteMediaItem(id uint) error {
return db.Unscoped().Delete(&model.MediaItem{}, id).Error
}
// ClearMediaItems 硬删除指定类型的所有媒体条目(真正从数据库删除)
func ClearMediaItems(mediaType model.MediaType) error {
return db.Unscoped().Where("media_type = ?", mediaType).Delete(&model.MediaItem{}).Error
}
// ClearMediaItemsByScanPath 硬删除指定扫描路径的所有媒体条目
func ClearMediaItemsByScanPath(scanPathID uint) error {
return db.Unscoped().Where("scan_path_id = ?", scanPathID).Delete(&model.MediaItem{}).Error
}
// ClearMediaScrapedData 清空指定类型的所有刮削数据(保留扫描出的文件记录本身)
// 仅清空刮削结果相关字段,不删除条目,便于重新刮削。
// mediaType 为空时表示对所有类型生效。
func ClearMediaScrapedData(mediaType model.MediaType) (int64, error) {
tx := db.Unscoped().Model(&model.MediaItem{})
if mediaType != "" {
tx = tx.Where("media_type = ?", mediaType)
}
updates := map[string]interface{}{
"scraped_name": "",
"description": "",
"cover": "",
"release_date": "",
"rating": 0,
"genre": "",
"authors": "",
"plot": "",
"reviews": "",
"external_id": "",
"album_artist": "",
"publisher": "",
"isbn": "",
"lyrics": "",
"scraped_at": nil,
}
result := tx.Updates(updates)
return result.RowsAffected, result.Error
}
// ListAllValidMediaItemsForCheck 列出指定类型下所有未删除的媒体条目(仅取必要字段,用于失效检查)
// mediaType 为空时返回所有类型条目。
func ListAllValidMediaItemsForCheck(mediaType model.MediaType) ([]model.MediaItem, error) {
var items []model.MediaItem
tx := db.Model(&model.MediaItem{}).
Select("id, media_type, scan_path_id, file_name, folder_path, is_folder")
if mediaType != "" {
tx = tx.Where("media_type = ?", mediaType)
}
err := tx.Find(&items).Error
return items, err
}
// DeleteMediaItemsByIDs 按ID列表硬删除媒体条目
func DeleteMediaItemsByIDs(ids []uint) error {
if len(ids) == 0 {
return nil
}
return db.Unscoped().Where("id IN ?", ids).Delete(&model.MediaItem{}).Error
}
// ListAlbums 列出所有专辑(音乐专用)
func ListAlbums(q MediaItemQuery) ([]AlbumInfo, int64, error) {
type albumRow struct {
AlbumName string
AlbumArtist string
Cover string
ReleaseDate string
TrackCount int
ScanPathID uint
}
// 构建基础查询
baseQuery := db.Model(&model.MediaItem{}).
Where("media_type = ?", model.MediaTypeMusic)
if q.Hidden != nil {
baseQuery = baseQuery.Where("hidden = ?", *q.Hidden)
}
if q.ScanPathID > 0 {
baseQuery = baseQuery.Where("scan_path_id = ?", q.ScanPathID)
}
if q.Keyword != "" {
like := "%" + q.Keyword + "%"
baseQuery = baseQuery.Where("album_name LIKE ? OR album_artist LIKE ? OR scraped_name LIKE ?", like, like, like)
}
// 统计分组数(用子查询)
var total int64
if err := db.Table("(?) as sub", baseQuery.
Select("album_name, album_artist").
Group("album_name, album_artist")).
Count(&total).Error; err != nil {
return nil, 0, err
}
if q.PageSize <= 0 {
q.PageSize = 20
}
if q.Page <= 0 {
q.Page = 1
}
tx := baseQuery.
Select("album_name, album_artist, MAX(cover) as cover, MAX(release_date) as release_date, COUNT(*) as track_count, MAX(scan_path_id) as scan_path_id").
Group("album_name, album_artist").
Offset((q.Page - 1) * q.PageSize).Limit(q.PageSize)
var rows []albumRow
if err := tx.Scan(&rows).Error; err != nil {
return nil, 0, err
}
albums := make([]AlbumInfo, len(rows))
for i, r := range rows {
albums[i] = AlbumInfo{
AlbumName: r.AlbumName,
AlbumArtist: r.AlbumArtist,
Cover: r.Cover,
ReleaseDate: r.ReleaseDate,
TrackCount: r.TrackCount,
ScanPathID: r.ScanPathID,
}
}
return albums, total, nil
}
// AlbumInfo 专辑信息
type AlbumInfo struct {
AlbumName string `json:"album_name"`
AlbumArtist string `json:"album_artist"`
Cover string `json:"cover"`
ReleaseDate string `json:"release_date"`
TrackCount int `json:"track_count"`
ScanPathID uint `json:"scan_path_id"`
}
// GetAlbumTracks 获取专辑曲目列表
// 支持两种模式:
// 1. 普通模式(is_folder=false):直接返回独立文件记录
// 2. 合并文件夹模式(is_folder=true):把 episodes 展开成虚拟 MediaItem 列表
// 展开后每条记录的 folder_path = 原folder_path/file_name(文件夹实际路径),file_name = episode.FileName
func GetAlbumTracks(albumName, albumArtist string) ([]model.MediaItem, error) {
var items []model.MediaItem
tx := db.Where("media_type = ?", model.MediaTypeMusic)
if albumName != "" {
tx = tx.Where("album_name = ?", albumName)
} else {
tx = tx.Where("(album_name = '' OR album_name IS NULL)")
}
if albumArtist != "" {
tx = tx.Where("album_artist = ?", albumArtist)
}
err := tx.Order("track_number asc").Find(&items).Error
if err != nil {
return nil, err
}
// 展开合并文件夹条目的 episodes
var result []model.MediaItem
for _, item := range items {
if !item.IsFolder || item.Episodes == "" {
result = append(result, item)
continue
}
// 解析 episodes
type EpisodeInfo struct {
FileName string `json:"file_name"`
Index int `json:"index"`
Title string `json:"title"`
}
var eps []EpisodeInfo
if err := json.Unmarshal([]byte(item.Episodes), &eps); err != nil || len(eps) == 0 {
// 解析失败则跳过该条目(不返回文件夹本身,避免播放路径错误)
continue
}
// 文件夹实际路径 = folder_path + "/" + file_name
actualDir := strings.TrimRight(item.FolderPath, "/") + "/" + item.FileName
for _, ep := range eps {
track := item // 复制基础信息(封面、专辑名、艺术家等)
track.ID = 0
track.IsFolder = false
track.FolderPath = actualDir
track.FileName = ep.FileName
track.TrackNumber = ep.Index
track.ScrapedName = ep.Title
track.Episodes = ""
result = append(result, track)
}
}
return result, nil
}
// ListFolderPaths 列出指定媒体类型下的所有文件夹路径(目录浏览模式)
func ListFolderPaths(mediaType model.MediaType) ([]string, error) {
var paths []string
err := db.Model(&model.MediaItem{}).
Where("media_type = ?", mediaType).
Distinct("folder_path").
Pluck("folder_path", &paths).Error
return paths, err
}
// GetUnscrappedItems 获取未刮削或刮削不完整的媒体条目
// 只要 scraped_at 为空,或 cover/scraped_name/plot 任一为空,就需要重新刮削
// limit <= 0 表示不限制数量,返回所有未刮削条目
func GetUnscrappedItems(mediaType model.MediaType, limit int) ([]model.MediaItem, error) {
var items []model.MediaItem
tx := db.Where(
"media_type = ? AND (scraped_at IS NULL OR cover = '' OR cover IS NULL OR scraped_name = '' OR scraped_name IS NULL OR plot = '' OR plot IS NULL)",
mediaType,
)
if limit > 0 {
tx = tx.Limit(limit)
}
err := tx.Find(&items).Error
return items, err
}
// ==================== 导入/导出 ====================
// ListAllMediaItems 列出指定媒体类型下的所有条目(不分页,用于导出)
// mediaType 为空时返回全部类型的所有条目
func ListAllMediaItems(mediaType model.MediaType) ([]model.MediaItem, error) {
var items []model.MediaItem
tx := db.Model(&model.MediaItem{})
if mediaType != "" {
tx = tx.Where("media_type = ?", mediaType)
}
err := tx.Order("id asc").Find(&items).Error
return items, err
}
// ListMediaItemsByScanPath 列出指定扫描路径下的所有条目(不分页,用于导出)
func ListMediaItemsByScanPath(scanPathID uint) ([]model.MediaItem, error) {
var items []model.MediaItem
err := db.Where("scan_path_id = ?", scanPathID).Order("id asc").Find(&items).Error
return items, err
}
// ListAllMediaScanPaths 列出所有扫描路径(不区分类型,用于全量导出)
func ListAllMediaScanPaths() ([]model.MediaScanPath, error) {
var paths []model.MediaScanPath
err := db.Order("id asc").Find(&paths).Error
return paths, err
}
// ImportMediaItems 批量导入媒体条目
// 策略:按 (folder_path, file_name, album_name) 唯一键 upsert:
// - 已存在则覆盖刮削字段(导入是用户主动行为,覆盖优先于保留)
// - 不存在则新建
//
// 注意:导入数据中的 ID/CreatedAt/UpdatedAt 会被忽略,由数据库重新分配,
// 避免和现有记录的主键冲突。
func ImportMediaItems(items []model.MediaItem, scanPathIDOverride *uint) (created, updated int, err error) {
for i := range items {
it := items[i]
// 清空主键和时间戳,让数据库重新生成
it.ID = 0
it.CreatedAt = time.Time{}
it.UpdatedAt = time.Time{}
it.DeletedAt = gorm.DeletedAt{}
// 如果指定了覆盖的 scan_path_id(按扫描路径导入场景),则强制覆盖
if scanPathIDOverride != nil {
it.ScanPathID = *scanPathIDOverride
}
var existing model.MediaItem
result := db.Unscoped().Where(
"folder_path = ? AND file_name = ? AND album_name = ?",
it.FolderPath, it.FileName, it.AlbumName,
).First(&existing)
if result.Error == gorm.ErrRecordNotFound {
if e := db.Create(&it).Error; e != nil {
return created, updated, e
}
created++
continue
}
if result.Error != nil {
return created, updated, result.Error
}
// 覆盖更新
it.ID = existing.ID
it.CreatedAt = existing.CreatedAt
it.DeletedAt = gorm.DeletedAt{}
if e := db.Unscoped().Save(&it).Error; e != nil {
return created, updated, e
}
updated++
}
return created, updated, nil
}
// ImportMediaScanPaths 批量导入扫描路径
// 策略:按 (media_type, path) 唯一对去重:
// - 已存在则更新名称/标签等可编辑字段
// - 不存在则新建并返回新ID
//
// 返回值 idMap:导入数据中的原始ID -> 数据库实际ID 的映射,
// 供随后的 MediaItem 导入用于把 scan_path_id 重新指向新ID。
func ImportMediaScanPaths(paths []model.MediaScanPath) (idMap map[uint]uint, created, updated int, err error) {
idMap = make(map[uint]uint, len(paths))
for i := range paths {
p := paths[i]
originalID := p.ID
p.ID = 0
p.CreatedAt = time.Time{}
p.UpdatedAt = time.Time{}
p.DeletedAt = gorm.DeletedAt{}
var existing model.MediaScanPath
result := db.Where("media_type = ? AND path = ?", p.MediaType, p.Path).First(&existing)
if result.Error == gorm.ErrRecordNotFound {
if e := db.Create(&p).Error; e != nil {
return idMap, created, updated, e
}
idMap[originalID] = p.ID
created++
continue
}
if result.Error != nil {
return idMap, created, updated, result.Error
}
// 已存在:更新可编辑字段
existing.Name = p.Name
existing.PathMerge = p.PathMerge
existing.TypeTag = p.TypeTag
existing.ContentTags = p.ContentTags
existing.EnableScrape = p.EnableScrape
if e := db.Save(&existing).Error; e != nil {
return idMap, created, updated, e
}
idMap[originalID] = existing.ID
updated++
}
return idMap, created, updated, nil
}
-170
View File
@@ -1,170 +0,0 @@
package fuse
import "github.com/winfsp/cgofuse/fuse"
type Fs struct {
RootFolder string
fuse.FileSystemBase
}
func (fs *Fs) Init() {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Destroy() {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Statfs(path string, stat *fuse.Statfs_t) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Mknod(path string, mode uint32, dev uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Mkdir(path string, mode uint32) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Unlink(path string) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Rmdir(path string) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Link(oldpath string, newpath string) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Symlink(target string, newpath string) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Readlink(path string) (int, string) {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Rename(oldpath string, newpath string) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Chmod(path string, mode uint32) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Chown(path string, uid uint32, gid uint32) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Utimens(path string, tmsp []fuse.Timespec) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Access(path string, mask uint32) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Create(path string, flags int, mode uint32) (int, uint64) {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Open(path string, flags int) (int, uint64) {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Getattr(path string, stat *fuse.Stat_t, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Truncate(path string, size int64, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Read(path string, buff []byte, ofst int64, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Write(path string, buff []byte, ofst int64, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Flush(path string, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Release(path string, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Fsync(path string, datasync bool, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Opendir(path string) (int, uint64) {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Readdir(path string, fill func(name string, stat *fuse.Stat_t, ofst int64) bool, ofst int64, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Releasedir(path string, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Fsyncdir(path string, datasync bool, fh uint64) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Setxattr(path string, name string, value []byte, flags int) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Getxattr(path string, name string) (int, []byte) {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Removexattr(path string, name string) int {
//TODO implement me
panic("implement me")
}
func (fs *Fs) Listxattr(path string, fill func(name string) bool) int {
//TODO implement me
panic("implement me")
}
var _ fuse.FileSystemInterface = (*Fs)(nil)
-9
View File
@@ -1,9 +0,0 @@
package fuse
import "github.com/winfsp/cgofuse/fuse"
func Mount(mountSrc, mountDst string, opts []string) {
fs := &Fs{RootFolder: mountSrc}
host := fuse.NewFileSystemHost(fs)
go host.Mount(mountDst, opts)
}
-663
View File
@@ -1,663 +0,0 @@
package media
import (
"encoding/binary"
"io"
"strconv"
"strings"
)
// MusicTag 音频文件标签信息
type MusicTag struct {
Title string // TIT2
Artist string // TPE1
Album string // TALB
AlbumArtist string // TPE2
TrackNumber int // TRCK
Year string // TYER / TDRC
Genre string // TCON
CoverData []byte // 封面图片原始字节(APIC / PICTURE)
CoverMIME string // 封面图片 MIME 类型(如 image/jpeg)
Lyrics string // 歌词内容(LRC 文本或纯文本):来自 USLT / SYLT / Vorbis LYRICS
}
// ParseID3v2 从 io.Reader 中解析 ID3v2 标签(只读取文件头部,不需要 Seek)
// 支持 ID3v2.3 和 ID3v2.4
func ParseID3v2(r io.Reader) (*MusicTag, error) {
// 读取 ID3v2 头部(10 字节)
header := make([]byte, 10)
if _, err := io.ReadFull(r, header); err != nil {
return nil, err
}
// 检查魔数
if string(header[0:3]) != "ID3" {
return nil, nil // 不是 ID3v2 文件,不报错
}
version := header[3] // 主版本号:3 = ID3v2.3, 4 = ID3v2.4
if version < 3 || version > 4 {
return nil, nil // 只支持 v2.3 和 v2.4
}
// 解析标签总大小(syncsafe integer,4 字节,每字节最高位为 0)
tagSize := syncsafeToInt(header[6:10])
if tagSize <= 0 || tagSize > 10*1024*1024 { // 最大 10MB
return nil, nil
}
// 读取所有帧数据
data := make([]byte, tagSize)
if _, err := io.ReadFull(r, data); err != nil {
return nil, nil // 读取失败时静默跳过
}
tag := &MusicTag{}
pos := 0
// 跳过扩展头部(如果有)
flags := header[5]
if flags&0x40 != 0 { // 有扩展头部
if pos+4 > len(data) {
return tag, nil
}
extSize := int(binary.BigEndian.Uint32(data[pos : pos+4]))
pos += extSize
}
// 解析帧
for pos+10 <= len(data) {
frameID := string(data[pos : pos+4])
if frameID == "\x00\x00\x00\x00" {
break // 填充区域,结束
}
frameSize := int(binary.BigEndian.Uint32(data[pos+4 : pos+8]))
pos += 10 // 跳过帧头(4+4+2)
if frameSize <= 0 || pos+frameSize > len(data) {
break
}
frameData := data[pos : pos+frameSize]
pos += frameSize
// 解析文本帧(T 开头的帧)
if len(frameID) == 4 && frameID[0] == 'T' && len(frameData) > 0 {
text := decodeID3Text(frameData)
switch frameID {
case "TIT2":
tag.Title = text
case "TPE1":
tag.Artist = text
case "TALB":
tag.Album = text
case "TPE2":
tag.AlbumArtist = text
case "TRCK":
// 格式可能是 "1" 或 "1/12"
parts := strings.SplitN(text, "/", 2)
if n, err := strconv.Atoi(strings.TrimSpace(parts[0])); err == nil {
tag.TrackNumber = n
}
case "TYER", "TDRC":
if tag.Year == "" {
tag.Year = text
}
case "TCON":
tag.Genre = parseID3Genre(text)
}
}
// 解析 APIC 帧(内嵌封面图片),只取第一张
if frameID == "APIC" && tag.CoverData == nil && len(frameData) > 1 {
tag.CoverData, tag.CoverMIME = parseAPICFrame(frameData)
}
// 解析 USLT 帧(非同步歌词)
// 格式:编码字节(1) + 语言(3字节 ASCII) + 描述字符串(null 结尾) + 歌词文本
if frameID == "USLT" && tag.Lyrics == "" && len(frameData) > 5 {
tag.Lyrics = parseUSLTFrame(frameData)
}
// 解析 SYLT 帧(同步歌词,二进制时间戳格式),仅在没有 USLT 时尝试转 LRC
if frameID == "SYLT" && tag.Lyrics == "" && len(frameData) > 6 {
if lrc := parseSYLTFrame(frameData); lrc != "" {
tag.Lyrics = lrc
}
}
}
return tag, nil
}
// parseUSLTFrame 解析 ID3v2 USLT 帧(非同步歌词)
// 格式:编码字节(1) + 语言(3字节 ASCII) + 描述字符串(null 结尾) + 歌词文本
func parseUSLTFrame(data []byte) string {
if len(data) < 5 {
return ""
}
encoding := data[0]
// 跳过编码(1) + 语言(3) = 4 字节
pos := 4
// 跳过描述字符串
if encoding == 0x01 || encoding == 0x02 {
// UTF-16:找 \x00\x00 结尾
for pos+1 < len(data) {
if data[pos] == 0 && data[pos+1] == 0 {
pos += 2
break
}
pos += 2
}
} else {
// ISO-8859-1 / UTF-8:找单个 \x00 结尾
for pos < len(data) {
if data[pos] == 0 {
pos++
break
}
pos++
}
}
if pos >= len(data) {
return ""
}
// 剩余字节就是歌词文本,按编码解码(复用 decodeID3Text 但需要把 encoding 字节加回头部)
lyricsRaw := make([]byte, 1+len(data)-pos)
lyricsRaw[0] = encoding
copy(lyricsRaw[1:], data[pos:])
return strings.TrimSpace(decodeID3Text(lyricsRaw))
}
// parseSYLTFrame 解析 ID3v2 SYLT 帧(同步歌词),转换为 LRC 格式
// 格式:编码(1) + 语言(3) + 时间戳格式(1) + 内容类型(1) + 描述(null 结尾) + 多组 (歌词文本 + null + 4字节时间戳)
// 时间戳格式:1=MPEG帧数(不支持),2=毫秒
func parseSYLTFrame(data []byte) string {
if len(data) < 6 {
return ""
}
encoding := data[0]
timestampFormat := data[4] // 1=MPEG帧, 2=ms
if timestampFormat != 2 {
return "" // 只支持毫秒时间戳
}
pos := 6 // 跳过 encoding(1) + lang(3) + timestamp_fmt(1) + content_type(1)
// 跳过描述字符串
if encoding == 0x01 || encoding == 0x02 {
for pos+1 < len(data) {
if data[pos] == 0 && data[pos+1] == 0 {
pos += 2
break
}
pos += 2
}
} else {
for pos < len(data) {
if data[pos] == 0 {
pos++
break
}
pos++
}
}
var sb strings.Builder
wide := encoding == 0x01 || encoding == 0x02
for pos < len(data) {
// 找文本终止符
textStart := pos
textEnd := -1
if wide {
for i := pos; i+1 < len(data); i += 2 {
if data[i] == 0 && data[i+1] == 0 {
textEnd = i
break
}
}
} else {
for i := pos; i < len(data); i++ {
if data[i] == 0 {
textEnd = i
break
}
}
}
if textEnd < 0 {
break
}
// 解码歌词文本
lineRaw := make([]byte, 1+textEnd-textStart)
lineRaw[0] = encoding
copy(lineRaw[1:], data[textStart:textEnd])
lineText := decodeID3Text(lineRaw)
// 跳过文本终止符
if wide {
pos = textEnd + 2
} else {
pos = textEnd + 1
}
// 4 字节时间戳(毫秒)
if pos+4 > len(data) {
break
}
ms := int(binary.BigEndian.Uint32(data[pos : pos+4]))
pos += 4
minutes := ms / 60000
seconds := (ms % 60000) / 1000
cs := (ms % 1000) / 10
// LRC 标准格式 [mm:ss.xx]
sb.WriteString("[")
if minutes < 10 {
sb.WriteString("0")
}
sb.WriteString(strconv.Itoa(minutes))
sb.WriteString(":")
if seconds < 10 {
sb.WriteString("0")
}
sb.WriteString(strconv.Itoa(seconds))
sb.WriteString(".")
if cs < 10 {
sb.WriteString("0")
}
sb.WriteString(strconv.Itoa(cs))
sb.WriteString("]")
sb.WriteString(lineText)
sb.WriteString("\n")
}
return sb.String()
}
// parseAPICFrame 解析 ID3v2 APIC 帧,返回图片字节和 MIME 类型
// APIC 格式:编码字节(1) + MIME字符串(null结尾) + 图片类型(1) + 描述字符串(null结尾) + 图片数据
func parseAPICFrame(data []byte) ([]byte, string) {
if len(data) < 4 {
return nil, ""
}
// encoding := data[0] // 文本编码(只影响描述字段,MIME 始终是 ASCII)
pos := 1
// 读取 MIME 类型(null 结尾的 ASCII 字符串)
nullIdx := -1
for i := pos; i < len(data); i++ {
if data[i] == 0 {
nullIdx = i
break
}
}
if nullIdx < 0 {
return nil, ""
}
mimeType := string(data[pos:nullIdx])
pos = nullIdx + 1
if pos >= len(data) {
return nil, ""
}
// 图片类型(1字节,3 = Cover (front),但我们取任意第一张)
pos++ // 跳过图片类型字节
// 跳过描述字符串(null 结尾,编码由第一字节决定)
encoding := data[0]
if encoding == 0x01 || encoding == 0x02 {
// UTF-16:找 \x00\x00 结尾
for pos+1 < len(data) {
if data[pos] == 0 && data[pos+1] == 0 {
pos += 2
break
}
pos += 2
}
} else {
// ISO-8859-1 / UTF-8:找单个 \x00 结尾
for pos < len(data) {
if data[pos] == 0 {
pos++
break
}
pos++
}
}
if pos >= len(data) {
return nil, ""
}
// 标准化 MIME 类型
if mimeType == "" || mimeType == "image/" {
mimeType = "image/jpeg" // 默认
}
imgData := make([]byte, len(data)-pos)
copy(imgData, data[pos:])
return imgData, mimeType
}
// syncsafeToInt 将 syncsafe integer 转换为普通整数
func syncsafeToInt(b []byte) int {
result := 0
for _, v := range b {
result = (result << 7) | int(v&0x7F)
}
return result
}
// decodeID3Text 解码 ID3 文本帧(第一字节是编码标识)
func decodeID3Text(data []byte) string {
if len(data) == 0 {
return ""
}
encoding := data[0]
content := data[1:]
// 去掉末尾的 null 字节
switch encoding {
case 0x00: // ISO-8859-1
// 去掉末尾 null
content = trimNull(content, false)
// 尝试 UTF-8 解码,如果失败则按 Latin-1 处理
return latin1ToUTF8(content)
case 0x01: // UTF-16 with BOM
content = trimNull(content, true)
return utf16ToUTF8(content)
case 0x02: // UTF-16 BE without BOM
content = trimNull(content, true)
return utf16BEToUTF8(content)
case 0x03: // UTF-8
content = trimNull(content, false)
return string(content)
default:
return string(content)
}
}
// trimNull 去掉末尾的 null 字节
func trimNull(b []byte, wide bool) []byte {
if wide {
// UTF-16: 去掉末尾的 \x00\x00
for len(b) >= 2 && b[len(b)-2] == 0 && b[len(b)-1] == 0 {
b = b[:len(b)-2]
}
} else {
for len(b) > 0 && b[len(b)-1] == 0 {
b = b[:len(b)-1]
}
}
return b
}
// latin1ToUTF8 将 Latin-1 编码转换为 UTF-8
func latin1ToUTF8(b []byte) string {
runes := make([]rune, len(b))
for i, v := range b {
runes[i] = rune(v)
}
return string(runes)
}
// utf16ToUTF8 将 UTF-16(带 BOM)转换为 UTF-8
func utf16ToUTF8(b []byte) string {
if len(b) < 2 {
return ""
}
var bigEndian bool
if b[0] == 0xFF && b[1] == 0xFE {
bigEndian = false
b = b[2:]
} else if b[0] == 0xFE && b[1] == 0xFF {
bigEndian = true
b = b[2:]
}
return decodeUTF16(b, bigEndian)
}
// utf16BEToUTF8 将 UTF-16 BE(无 BOM)转换为 UTF-8
func utf16BEToUTF8(b []byte) string {
return decodeUTF16(b, true)
}
// decodeUTF16 解码 UTF-16 字节序列
func decodeUTF16(b []byte, bigEndian bool) string {
if len(b)%2 != 0 {
b = b[:len(b)-1]
}
runes := make([]rune, 0, len(b)/2)
for i := 0; i+1 < len(b); i += 2 {
var code uint16
if bigEndian {
code = uint16(b[i])<<8 | uint16(b[i+1])
} else {
code = uint16(b[i+1])<<8 | uint16(b[i])
}
runes = append(runes, rune(code))
}
return string(runes)
}
// parseID3Genre 解析 ID3 流派字段(可能是 "(17)" 格式的数字引用)
func parseID3Genre(s string) string {
s = strings.TrimSpace(s)
if len(s) > 2 && s[0] == '(' && s[len(s)-1] == ')' {
// 数字引用格式,直接返回原始字符串
return s
}
return s
}
// ─── FLAC Vorbis Comment 解析 ────────────────────────────────────────────────
// ParseFLACVorbisComment 从 io.Reader 中解析 FLAC 文件的 Vorbis Comment 元数据
// FLAC 格式:4字节魔数 "fLaC" + 若干 METADATA_BLOCK
// METADATA_BLOCK:1字节(最高位=是否最后块, 低7位=块类型) + 3字节长度 + 数据
// 块类型 4 = VORBIS_COMMENT
func ParseFLACVorbisComment(r io.Reader) (*MusicTag, error) {
// 读取魔数(4字节)
magic := make([]byte, 4)
if _, err := io.ReadFull(r, magic); err != nil {
return nil, err
}
if string(magic) != "fLaC" {
return nil, nil // 不是 FLAC 文件
}
// 遍历 METADATA_BLOCK,找到 VORBIS_COMMENT(类型4)和 PICTURE(类型6)
var tag *MusicTag
for {
// 读取块头(4字节:1字节标志+类型 + 3字节长度)
blockHeader := make([]byte, 4)
if _, err := io.ReadFull(r, blockHeader); err != nil {
return nil, nil // 读取失败,静默跳过
}
isLast := blockHeader[0]&0x80 != 0
blockType := blockHeader[0] & 0x7F
blockLen := int(blockHeader[1])<<16 | int(blockHeader[2])<<8 | int(blockHeader[3])
if blockLen < 0 || blockLen > 16*1024*1024 { // 最大 16MB
return nil, nil
}
if blockType == 4 {
// VORBIS_COMMENT 块
data := make([]byte, blockLen)
if _, err := io.ReadFull(r, data); err != nil {
return nil, nil
}
tag = parseVorbisCommentData(data)
if isLast {
return tag, nil
}
// 继续读取后续块,寻找 PICTURE 块(类型6)
continue
}
if blockType == 6 {
// PICTURE 块(FLAC 内嵌封面)
data := make([]byte, blockLen)
if _, err := io.ReadFull(r, data); err == nil && tag != nil && tag.CoverData == nil {
tag.CoverData, tag.CoverMIME = parseFLACPictureBlock(data)
} else if err != nil {
// 读取失败,跳过
_ = err
}
if isLast {
return tag, nil
}
continue
}
// 跳过此块
if _, err := io.CopyN(io.Discard, r, int64(blockLen)); err != nil {
return nil, nil
}
if isLast {
break
}
}
return tag, nil
}
// parseFLACPictureBlock 解析 FLAC PICTURE 元数据块,返回图片字节和 MIME 类型
// 格式(大端序):
// 4字节 picture_type
// 4字节 mime_length + mime_string
// 4字节 description_length + description_string
// 4字节 width, 4字节 height, 4字节 color_depth, 4字节 color_count
// 4字节 data_length + data
func parseFLACPictureBlock(data []byte) ([]byte, string) {
if len(data) < 8 {
return nil, ""
}
pos := 4 // 跳过 picture_type
// 读取 MIME 类型
if pos+4 > len(data) {
return nil, ""
}
mimeLen := int(binary.BigEndian.Uint32(data[pos : pos+4]))
pos += 4
if pos+mimeLen > len(data) {
return nil, ""
}
mimeType := string(data[pos : pos+mimeLen])
pos += mimeLen
// 跳过描述字符串
if pos+4 > len(data) {
return nil, ""
}
descLen := int(binary.BigEndian.Uint32(data[pos : pos+4]))
pos += 4 + descLen
// 跳过 width(4) + height(4) + color_depth(4) + color_count(4)
pos += 16
// 读取图片数据
if pos+4 > len(data) {
return nil, ""
}
dataLen := int(binary.BigEndian.Uint32(data[pos : pos+4]))
pos += 4
if pos+dataLen > len(data) {
return nil, ""
}
if mimeType == "" {
mimeType = "image/jpeg"
}
imgData := make([]byte, dataLen)
copy(imgData, data[pos:pos+dataLen])
return imgData, mimeType
}
// parseVorbisCommentData 解析 Vorbis Comment 数据块
// 格式(小端序):
// 4字节 vendor_length + vendor_string
// 4字节 comment_count
// 每条注释:4字节长度 + UTF-8字符串(格式 "KEY=VALUE")
func parseVorbisCommentData(data []byte) *MusicTag {
tag := &MusicTag{}
pos := 0
// 跳过 vendor string
if pos+4 > len(data) {
return tag
}
vendorLen := int(binary.LittleEndian.Uint32(data[pos : pos+4]))
pos += 4 + vendorLen
// 读取注释数量
if pos+4 > len(data) {
return tag
}
commentCount := int(binary.LittleEndian.Uint32(data[pos : pos+4]))
pos += 4
for i := 0; i < commentCount; i++ {
if pos+4 > len(data) {
break
}
commentLen := int(binary.LittleEndian.Uint32(data[pos : pos+4]))
pos += 4
if commentLen <= 0 || pos+commentLen > len(data) {
break
}
comment := string(data[pos : pos+commentLen])
pos += commentLen
// 解析 "KEY=VALUE" 格式(大小写不敏感)
eqIdx := strings.IndexByte(comment, '=')
if eqIdx < 0 {
continue
}
key := strings.ToUpper(comment[:eqIdx])
value := strings.TrimSpace(comment[eqIdx+1:])
if value == "" {
continue
}
switch key {
case "TITLE":
tag.Title = value
case "ARTIST":
if tag.Artist == "" {
tag.Artist = value
}
case "ALBUM":
tag.Album = value
case "ALBUMARTIST", "ALBUM ARTIST":
tag.AlbumArtist = value
case "TRACKNUMBER", "TRACK":
// 格式可能是 "1" 或 "1/12"
parts := strings.SplitN(value, "/", 2)
if n, err := strconv.Atoi(strings.TrimSpace(parts[0])); err == nil {
tag.TrackNumber = n
}
case "DATE", "YEAR":
if tag.Year == "" && len(value) >= 4 {
tag.Year = value[:4]
}
case "GENRE":
tag.Genre = value
case "LYRICS", "UNSYNCEDLYRICS", "UNSYNCED LYRICS", "SYNCEDLYRICS", "SYNCED LYRICS":
// FLAC/Vorbis 内嵌歌词字段(不同打标软件使用不同 key,全部兼容)
if tag.Lyrics == "" {
tag.Lyrics = value
}
}
}
return tag
}
-130
View File
@@ -1,130 +0,0 @@
package media
import (
"context"
"encoding/base64"
"encoding/json"
stdpath "path"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/fs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
)
// FillMusicFromLocal 在刮削前/失败时,使用音乐文件本身的元数据(ID3 / Vorbis Comment)
// 以及同目录 .lrc 歌词,**只填充 item 中为空的字段**。
//
// 适用场景:
// 1. 用户清空刮削后重新刮削,原本由扫描阶段填入的本地数据(封面/歌词等)已丢失
// 2. 在线刮削失败,退而求其次用文件本地信息补全
//
// 行为:
// - is_folder=true 的合并文件夹模式:使用 episodes 中第一首音乐文件读取
// - is_folder=false:使用 folder_path/file_name 直接读取
//
// 错误以静默忽略为主——本地兜底失败不应阻塞主流程。
func FillMusicFromLocal(ctx context.Context, item *model.MediaItem) {
if item == nil || item.MediaType != model.MediaTypeMusic {
return
}
musicPath := pickMusicFilePath(item)
if musicPath == "" {
return
}
// 解析 tag(mp3 / flac / 其它)
ext := strings.ToLower(stdpath.Ext(musicPath))
var tag *MusicTag
readCtx, readCancel := context.WithTimeout(ctx, 15*time.Second)
if reader := FetchFileReader(readCtx, musicPath); reader != nil {
switch ext {
case ".flac":
tag, _ = ParseFLACVorbisComment(reader)
default:
tag, _ = ParseID3v2(reader)
}
_ = reader.Close()
}
readCancel()
// 应用 tag 到空字段
if tag != nil {
if item.AlbumName == "" && tag.Album != "" {
item.AlbumName = tag.Album
}
if item.AlbumArtist == "" {
if tag.AlbumArtist != "" {
item.AlbumArtist = tag.AlbumArtist
} else if tag.Artist != "" {
item.AlbumArtist = tag.Artist
}
}
if item.ScrapedName == "" {
if tag.Title != "" {
item.ScrapedName = tag.Title
} else if item.AlbumName != "" {
item.ScrapedName = item.AlbumName
}
}
if item.TrackNumber == 0 && tag.TrackNumber > 0 {
item.TrackNumber = tag.TrackNumber
}
if item.ReleaseDate == "" && len(tag.Year) >= 4 {
item.ReleaseDate = tag.Year[:4] + "-01-01"
}
if item.Genre == "" && tag.Genre != "" {
item.Genre = tag.Genre
}
if item.Authors == "" && tag.Artist != "" {
if b, err := json.Marshal([]string{tag.Artist}); err == nil {
item.Authors = string(b)
}
}
if item.Cover == "" && len(tag.CoverData) > 0 {
mimeType := tag.CoverMIME
if mimeType == "" {
mimeType = "image/jpeg"
}
item.Cover = "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(tag.CoverData)
}
}
// 歌词:若为空则尝试 .lrc 同名文件 / 内嵌歌词
if item.Lyrics == "" {
if lrc := loadLyricsForMusic(ctx, musicPath, tag); lrc != "" {
item.Lyrics = lrc
}
}
}
// pickMusicFilePath 从 MediaItem 推断出可读取的音乐文件 VFS 路径
// 合并模式 (is_folder=true): 取 episodes 中第一首;解析失败回退到 folder_path/file_name
// 普通模式: folder_path/file_name
func pickMusicFilePath(item *model.MediaItem) string {
if item == nil {
return ""
}
if item.IsFolder {
// episodes 是 JSON 数组:[{file_name,index,title,...}]
if item.Episodes != "" {
var eps []EpisodeInfo
if err := json.Unmarshal([]byte(item.Episodes), &eps); err == nil && len(eps) > 0 {
// 文件夹模式下 folder_path 是扫描根,file_name 是文件夹名
return stdpath.Join(item.FolderPath, item.FileName, eps[0].FileName)
}
}
// 回退:尝试列出文件夹内第一首音乐
folder := stdpath.Join(item.FolderPath, item.FileName)
if entries, err := fs.List(context.Background(), folder, &fs.ListArgs{NoLog: true}); err == nil {
for _, e := range entries {
if !e.IsDir() && isMediaFile(e.GetName(), model.MediaTypeMusic) {
return stdpath.Join(folder, e.GetName())
}
}
}
return ""
}
return stdpath.Join(item.FolderPath, item.FileName)
}
-736
View File
@@ -1,736 +0,0 @@
package media
import (
"context"
"encoding/base64"
"encoding/json"
"io"
"mime"
stdpath "path"
"regexp"
"strings"
"sync"
"time"
"unicode"
log "github.com/sirupsen/logrus"
"github.com/OpenListTeam/OpenList/v4/internal/db"
"github.com/OpenListTeam/OpenList/v4/internal/fs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/http_range"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
)
// 支持的文件扩展名
var (
videoExts = map[string]bool{
".mp4": true, ".mkv": true, ".avi": true, ".mov": true,
".wmv": true, ".flv": true, ".webm": true, ".m4v": true,
".ts": true, ".rmvb": true, ".rm": true, ".3gp": true,
}
musicExts = map[string]bool{
".mp3": true, ".flac": true, ".aac": true, ".ogg": true,
".wav": true, ".wma": true, ".m4a": true, ".ape": true,
".opus": true, ".aiff": true,
}
imageExts = map[string]bool{
".jpg": true, ".jpeg": true, ".png": true, ".gif": true,
".webp": true, ".bmp": true, ".tiff": true, ".svg": true,
".heic": true, ".avif": true,
}
bookExts = map[string]bool{
".epub": true, ".pdf": true, ".mobi": true, ".azw3": true,
".txt": true, ".djvu": true, ".cbz": true, ".cbr": true,
}
)
// ScanProgress 扫描进度(全局,按媒体类型维护)
type ScanProgress struct {
mu sync.RWMutex
Running bool
Total int
Done int
Message string
Error string
}
var progressMap = map[model.MediaType]*ScanProgress{
model.MediaTypeVideo: {},
model.MediaTypeMusic: {},
model.MediaTypeImage: {},
model.MediaTypeBook: {},
}
// GetProgress 获取扫描进度
func GetProgress(mediaType model.MediaType) model.MediaScanProgress {
p, ok := progressMap[mediaType]
if !ok {
return model.MediaScanProgress{MediaType: mediaType}
}
p.mu.RLock()
defer p.mu.RUnlock()
return model.MediaScanProgress{
MediaType: mediaType,
Running: p.Running,
Total: p.Total,
Done: p.Done,
Message: p.Message,
Error: p.Error,
}
}
// ScanMedia 扫描媒体文件(异步),扫描指定媒体类型下的所有启用扫描路径
func ScanMedia(cfg *model.MediaConfig) {
p, ok := progressMap[cfg.MediaType]
if !ok {
return
}
p.mu.Lock()
if p.Running {
p.mu.Unlock()
return
}
p.Running = true
p.Total = 0
p.Done = 0
p.Error = ""
p.Message = "正在扫描..."
p.mu.Unlock()
go func() {
defer func() {
p.mu.Lock()
p.Running = false
p.mu.Unlock()
}()
// 获取该媒体类型下所有扫描路径
scanPaths, err := db.ListMediaScanPaths(cfg.MediaType)
if err != nil {
p.mu.Lock()
p.Error = err.Error()
p.Message = "获取扫描路径失败"
p.mu.Unlock()
return
}
if len(scanPaths) == 0 {
p.mu.Lock()
p.Message = "没有配置扫描路径"
p.mu.Unlock()
return
}
for i := range scanPaths {
sp := &scanPaths[i]
p.mu.Lock()
p.Message = "正在扫描路径: " + sp.Name
p.mu.Unlock()
if err := doScanPath(sp, p); err != nil {
log.Errorf("media scan error [%s] path[%s]: %v", cfg.MediaType, sp.Path, err)
} else {
// 更新最后扫描时间
now := time.Now()
sp.LastScanAt = &now
_ = db.UpdateMediaScanPath(sp)
}
}
p.mu.Lock()
p.Message = "扫描完成"
p.mu.Unlock()
// 更新媒体库最后扫描时间
now := time.Now()
cfg.LastScanAt = &now
_ = db.SaveMediaConfig(cfg)
}()
}
// ScanMediaPath 扫描单个扫描路径(异步)
func ScanMediaPath(sp *model.MediaScanPath) {
p, ok := progressMap[sp.MediaType]
if !ok {
return
}
p.mu.Lock()
if p.Running {
p.mu.Unlock()
return
}
p.Running = true
p.Total = 0
p.Done = 0
p.Error = ""
p.Message = "正在扫描路径: " + sp.Name
p.mu.Unlock()
go func() {
defer func() {
p.mu.Lock()
p.Running = false
p.mu.Unlock()
}()
if err := doScanPath(sp, p); err != nil {
p.mu.Lock()
p.Error = err.Error()
p.Message = "扫描失败"
p.mu.Unlock()
log.Errorf("media scan error [%s] path[%s]: %v", sp.MediaType, sp.Path, err)
} else {
p.mu.Lock()
p.Message = "扫描完成"
p.mu.Unlock()
now := time.Now()
sp.LastScanAt = &now
_ = db.UpdateMediaScanPath(sp)
}
}()
}
func doScanPath(sp *model.MediaScanPath, p *ScanProgress) error {
scanRoot := sp.Path
if scanRoot == "" {
scanRoot = "/"
}
// 收集所有待处理路径(VFS 虚拟路径)
var targets []string
ctx := context.Background()
if sp.PathMerge {
// 路径合并模式:
// - 子文件夹 → 作为一个合并条目(带 Episodes 选集信息)
// - 直接放在根目录下的单个媒体文件 → 正常作为独立条目扫描
entries, err := fs.List(ctx, scanRoot, &fs.ListArgs{NoLog: true, Refresh: true})
if err != nil {
return err
}
for _, e := range entries {
childPath := stdpath.Join(scanRoot, e.GetName())
if e.IsDir() {
targets = append(targets, childPath)
} else if isMediaFile(e.GetName(), sp.MediaType) {
targets = append(targets, childPath)
}
}
} else {
// 普通模式:递归扫描所有匹配文件(每个目录都刷新缓存)
if err := walkVFS(ctx, scanRoot, sp.MediaType, &targets); err != nil {
return err
}
}
p.mu.Lock()
p.Total += len(targets)
p.mu.Unlock()
// 音乐类型:按专辑合并处理
if sp.MediaType == model.MediaTypeMusic {
return doScanMusicByAlbum(ctx, targets, sp, p)
}
for _, target := range targets {
item, err := buildMediaItemFromVFS(ctx, target, sp)
if err != nil {
log.Warnf("build media item error [%s]: %v", target, err)
continue
}
// 路径合并模式下,扫描文件夹内的文件,填充选集信息
if sp.PathMerge && item.IsFolder {
if episodes, err := buildEpisodesFromFolder(ctx, target, sp.MediaType); err == nil {
item.Episodes = episodes
} else {
log.Warnf("build episodes error [%s]: %v", target, err)
}
}
if err := db.CreateOrUpdateMediaItem(item); err != nil {
log.Warnf("save media item error [%s]: %v", target, err)
}
p.mu.Lock()
p.Done++
p.Message = "已扫描: " + stdpath.Base(target)
p.mu.Unlock()
}
return nil
}
// doScanMusicByAlbum 音乐扫描
// 存储规则:
// - 普通文件(is_folder=false):
// folder_path=文件所在真实目录,file_name=音乐文件名,episodes=空
// - 合并文件夹(is_folder=true):
// folder_path=文件夹所在真实父目录,file_name=文件夹名,
// episodes=文件夹内所有音乐文件列表,封面/标签取第一首
func doScanMusicByAlbum(ctx context.Context, targets []string, sp *model.MediaScanPath, p *ScanProgress) error {
for _, target := range targets {
obj, err := fs.Get(ctx, target, &fs.GetArgs{NoLog: true})
if err != nil {
log.Warnf("get vfs object error [%s]: %v", target, err)
p.mu.Lock()
p.Done++
p.mu.Unlock()
continue
}
name := obj.GetName()
if obj.IsDir() {
// ---- 合并文件夹模式:文件夹条目 ----
// 列出文件夹内所有音乐文件
entries, err := fs.List(ctx, target, &fs.ListArgs{NoLog: true})
if err != nil {
log.Warnf("list music folder error [%s]: %v", target, err)
p.mu.Lock()
p.Done++
p.mu.Unlock()
continue
}
var musicFiles []string
var episodes []EpisodeInfo
for idx, e := range entries {
if e.IsDir() || !isMediaFile(e.GetName(), sp.MediaType) {
continue
}
musicFiles = append(musicFiles, stdpath.Join(target, e.GetName()))
ep := EpisodeInfo{
FileName: e.GetName(),
Index: idx + 1,
Title: strings.TrimSuffix(e.GetName(), stdpath.Ext(e.GetName())),
}
episodes = append(episodes, ep)
}
if len(musicFiles) == 0 {
p.mu.Lock()
p.Done++
p.mu.Unlock()
continue
}
// 取第一首音乐文件读取标签
firstFile := musicFiles[0]
firstExt := strings.ToLower(stdpath.Ext(firstFile))
var tag *MusicTag
readCtx, readCancel := context.WithTimeout(ctx, 15*time.Second)
if reader := FetchFileReader(readCtx, firstFile); reader != nil {
switch firstExt {
case ".flac":
tag, _ = ParseFLACVorbisComment(reader)
default:
tag, _ = ParseID3v2(reader)
}
_ = reader.Close()
}
readCancel()
// 解析标签(用文件夹名作为默认专辑名)
albumName := name
albumArtist := ""
cover := ""
releaseDate := ""
genre := ""
authors := ""
if tag != nil {
if tag.Album != "" {
albumName = tag.Album
}
albumArtist = tag.AlbumArtist
if albumArtist == "" {
albumArtist = tag.Artist
}
if tag.Year != "" && len(tag.Year) >= 4 {
releaseDate = tag.Year[:4] + "-01-01"
}
if tag.Genre != "" {
genre = tag.Genre
}
if tag.Artist != "" {
if authorsJSON, err := json.Marshal([]string{tag.Artist}); err == nil {
authors = string(authorsJSON)
}
}
if len(tag.CoverData) > 0 {
mimeType := tag.CoverMIME
if mimeType == "" {
mimeType = "image/jpeg"
}
cover = "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(tag.CoverData)
}
}
episodesJSON := ""
if b, err := json.Marshal(episodes); err == nil {
episodesJSON = string(b)
}
// 加载歌词:同目录同名 .lrc 优先,其次内嵌歌词
lyrics := loadLyricsForMusic(ctx, firstFile, tag)
item := &model.MediaItem{
MediaType: sp.MediaType,
ScanPathID: sp.ID,
FileName: name, // 文件夹名
FolderPath: stdpath.Dir(target), // 文件夹所在的真实父目录
IsFolder: true,
AlbumName: albumName,
AlbumArtist: albumArtist,
ScrapedName: albumName,
Cover: cover,
ReleaseDate: releaseDate,
Genre: genre,
Authors: authors,
Lyrics: lyrics,
Episodes: episodesJSON,
}
if err := db.CreateOrUpdateMediaItem(item); err != nil {
log.Warnf("save music folder item error [%s]: %v", target, err)
}
} else {
// ---- 普通文件:每首歌独立一条记录 ----
folderPath := stdpath.Dir(target)
ext := strings.ToLower(stdpath.Ext(name))
var tag *MusicTag
readCtx, readCancel := context.WithTimeout(ctx, 15*time.Second)
if reader := FetchFileReader(readCtx, target); reader != nil {
switch ext {
case ".flac":
tag, _ = ParseFLACVorbisComment(reader)
default:
tag, _ = ParseID3v2(reader)
}
_ = reader.Close()
}
readCancel()
albumName := ""
albumArtist := ""
trackTitle := strings.TrimSuffix(name, stdpath.Ext(name))
trackNumber := 0
cover := ""
releaseDate := ""
genre := ""
authors := ""
if tag != nil {
albumName = tag.Album
albumArtist = tag.AlbumArtist
if albumArtist == "" {
albumArtist = tag.Artist
}
if tag.Title != "" {
trackTitle = tag.Title
}
trackNumber = tag.TrackNumber
if tag.Year != "" && len(tag.Year) >= 4 {
releaseDate = tag.Year[:4] + "-01-01"
}
if tag.Genre != "" {
genre = tag.Genre
}
if tag.Artist != "" {
if authorsJSON, err := json.Marshal([]string{tag.Artist}); err == nil {
authors = string(authorsJSON)
}
}
if len(tag.CoverData) > 0 {
mimeType := tag.CoverMIME
if mimeType == "" {
mimeType = "image/jpeg"
}
cover = "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(tag.CoverData)
}
}
// 加载歌词:同目录同名 .lrc 优先,其次内嵌歌词
lyrics := loadLyricsForMusic(ctx, target, tag)
item := &model.MediaItem{
MediaType: sp.MediaType,
ScanPathID: sp.ID,
FileName: name,
FolderPath: folderPath,
IsFolder: false,
FileSize: obj.GetSize(),
MimeType: mime.TypeByExtension(ext),
AlbumName: albumName,
AlbumArtist: albumArtist,
ScrapedName: trackTitle,
Cover: cover,
ReleaseDate: releaseDate,
Genre: genre,
Authors: authors,
TrackNumber: trackNumber,
Lyrics: lyrics,
}
if err := db.CreateOrUpdateMediaItem(item); err != nil {
log.Warnf("save music item error [%s]: %v", target, err)
}
}
p.mu.Lock()
p.Done++
p.Message = "已扫描: " + name
p.mu.Unlock()
}
return nil
}
// loadLyricsForMusic 为音乐文件加载歌词:
// 1. 优先读取同目录同名 .lrc 文件(如 song.mp3 → song.lrc)
// 2. 如果不存在,使用 tag 中的内嵌歌词(USLT / Vorbis LYRICS)
//
// musicVfsPath 为音乐文件的 VFS 全路径;tag 可为 nil
func loadLyricsForMusic(ctx context.Context, musicVfsPath string, tag *MusicTag) string {
// 尝试同目录同名 .lrc
ext := stdpath.Ext(musicVfsPath)
if ext != "" {
lrcPath := strings.TrimSuffix(musicVfsPath, ext) + ".lrc"
lrcCtx, lrcCancel := context.WithTimeout(ctx, 8*time.Second)
reader := FetchFileReader(lrcCtx, lrcPath)
if reader != nil {
// 限制读取上限 1MB,避免异常文件压垮内存
data, err := io.ReadAll(io.LimitReader(reader, 1024*1024))
_ = reader.Close()
lrcCancel()
if err == nil && len(data) > 0 {
// 去掉可能的 UTF-8 BOM
if len(data) >= 3 && data[0] == 0xEF && data[1] == 0xBB && data[2] == 0xBF {
data = data[3:]
}
s := strings.TrimSpace(string(data))
if s != "" {
return s
}
}
} else {
lrcCancel()
}
}
// 回退到内嵌歌词
if tag != nil && tag.Lyrics != "" {
return tag.Lyrics
}
return ""
}
// FetchFileReader 通过 VFS 路径获取文件内容流(用于刮削器读取文件内容)
// 优先使用 RangeReader 直接读取(本地存储无需 HTTP),失败时回退到 HTTP URL
// 返回 nil 表示无法获取(不影响主流程)
func FetchFileReader(ctx context.Context, vfsPath string) io.ReadCloser {
link, file, err := fs.Link(ctx, vfsPath, model.LinkArgs{})
if err != nil || link == nil {
return nil
}
size := link.ContentLength
if size <= 0 {
size = file.GetSize()
}
rr, err := stream.GetRangeReaderFromLink(size, link)
if err != nil {
_ = link.Close()
return nil
}
rc, err := rr.RangeRead(ctx, http_range.Range{Length: size})
if err != nil {
_ = link.Close()
return nil
}
link.SyncClosers.Add(rc)
return utils.ReadCloser{Reader: rc, Closer: link}
}
// walkVFS 递归遍历 VFS 路径,收集匹配的媒体文件路径(每个目录都刷新缓存)
func walkVFS(ctx context.Context, dirPath string, mediaType model.MediaType, targets *[]string) error {
entries, err := fs.List(ctx, dirPath, &fs.ListArgs{NoLog: true, Refresh: true})
if err != nil {
log.Warnf("media scan: list vfs path [%s] error: %v", dirPath, err)
return nil // 跳过无权限目录,不中断整体扫描
}
for _, e := range entries {
childPath := stdpath.Join(dirPath, e.GetName())
if e.IsDir() {
if err := walkVFS(ctx, childPath, mediaType, targets); err != nil {
return err
}
} else if isMediaFile(e.GetName(), mediaType) {
*targets = append(*targets, childPath)
}
}
return nil
}
// buildMediaItemFromVFS 根据 VFS 路径构建 MediaItem(非音乐类型使用)
// 存储规则:
// - folder_path:文件/文件夹所在的真实父目录(VFS 父路径),用于前端按 folder_path + file_name 反查真实文件
// - file_name:文件夹就是文件夹名,文件就是文件名
// - episodes:文件夹时存里面每个文件的信息,文件时为空
func buildMediaItemFromVFS(ctx context.Context, vfsPath string, sp *model.MediaScanPath) (*model.MediaItem, error) {
obj, err := fs.Get(ctx, vfsPath, &fs.GetArgs{NoLog: true})
if err != nil {
return nil, err
}
name := obj.GetName()
ext := strings.ToLower(stdpath.Ext(name))
item := &model.MediaItem{
MediaType: sp.MediaType,
ScanPathID: sp.ID,
FileName: name, // 文件夹名 或 文件名
FolderPath: stdpath.Dir(vfsPath), // 真实父目录
IsFolder: obj.IsDir(),
ScrapedName: strings.TrimSuffix(name, stdpath.Ext(name)), // 去掉扩展名作为默认名称
}
if !obj.IsDir() {
// 普通文件:记录大小和 MIME 类型,episodes 为空
item.FileSize = obj.GetSize()
item.MimeType = mime.TypeByExtension(ext)
}
return item, nil
}
// episodeNumRe 匹配文件名开头的数字序号,支持 "1、" "2." "3-" "4 " 等分隔符
var episodeNumRe = regexp.MustCompile(`^(\d+)[、.\-\s_]+(.*)`)
// EpisodeInfo 选集信息
type EpisodeInfo struct {
FileName string `json:"file_name"` // 原始文件名(含扩展名)
Index int `json:"index"` // 序号,默认0,文件名开头有数字则取该数字
Title string `json:"title"` // 选集标题(去掉序号后的文件名,不含扩展名)
}
// buildEpisodesFromFolder 扫描文件夹内的媒体文件,构建选集信息 JSON 字符串
func buildEpisodesFromFolder(ctx context.Context, folderPath string, mediaType model.MediaType) (string, error) {
entries, err := fs.List(ctx, folderPath, &fs.ListArgs{NoLog: true})
if err != nil {
return "", err
}
var episodes []EpisodeInfo
for _, e := range entries {
if e.IsDir() {
continue
}
name := e.GetName()
if !isMediaFile(name, mediaType) {
continue
}
// 去掉扩展名得到裸文件名
ext := stdpath.Ext(name)
baseName := strings.TrimSuffix(name, ext)
ep := EpisodeInfo{
FileName: name,
Index: 0,
Title: baseName,
}
// 尝试从文件名开头提取数字序号
if m := episodeNumRe.FindStringSubmatch(baseName); len(m) == 3 {
if idx := parseLeadingInt(m[1]); idx > 0 {
ep.Index = idx
ep.Title = strings.TrimSpace(m[2])
}
} else {
// 文件名直接以纯数字开头(无分隔符),也尝试提取
if idx, rest := splitLeadingNumber(baseName); idx > 0 {
ep.Index = idx
ep.Title = strings.TrimSpace(rest)
}
}
episodes = append(episodes, ep)
}
if len(episodes) == 0 {
return "", nil
}
b, err := json.Marshal(episodes)
if err != nil {
return "", err
}
return string(b), nil
}
// parseLeadingInt 将纯数字字符串解析为 int,失败返回 0
func parseLeadingInt(s string) int {
v := 0
for _, c := range s {
if c < '0' || c > '9' {
return 0
}
v = v*10 + int(c-'0')
}
return v
}
// splitLeadingNumber 从字符串开头提取连续数字,返回 (数字值, 剩余字符串)
// 仅当开头确实有数字时才返回非零值
func splitLeadingNumber(s string) (int, string) {
i := 0
for i < len(s) && s[i] >= '0' && s[i] <= '9' {
i++
}
if i == 0 || i == len(s) {
// 没有数字,或者整个字符串都是数字(没有标题部分)
return 0, s
}
// 剩余部分必须以非字母数字字符开头,避免把 "1080p" 之类的误识别
if unicode.IsLetter(rune(s[i])) || unicode.IsDigit(rune(s[i])) {
return 0, s
}
v := parseLeadingInt(s[:i])
return v, s[i:]
}
// isMediaFile 判断文件名是否为指定媒体类型(按扩展名判断)
func isMediaFile(name string, mediaType model.MediaType) bool {
ext := strings.ToLower(stdpath.Ext(name))
switch mediaType {
case model.MediaTypeVideo:
return videoExts[ext]
case model.MediaTypeMusic:
return musicExts[ext]
case model.MediaTypeImage:
return imageExts[ext]
case model.MediaTypeBook:
return bookExts[ext]
}
return false
}
// GetSupportedExts 获取指定媒体类型支持的扩展名列表
func GetSupportedExts(mediaType model.MediaType) []string {
var extMap map[string]bool
switch mediaType {
case model.MediaTypeVideo:
extMap = videoExts
case model.MediaTypeMusic:
extMap = musicExts
case model.MediaTypeImage:
extMap = imageExts
case model.MediaTypeBook:
extMap = bookExts
default:
return nil
}
exts := make([]string, 0, len(extMap))
for ext := range extMap {
exts = append(exts, ext)
}
return exts
}
-402
View File
@@ -1,402 +0,0 @@
package scraper
import (
"archive/zip"
"bytes"
"encoding/base64"
"encoding/xml"
"image"
"image/jpeg"
"io"
"os"
"path/filepath"
"strings"
"time"
"github.com/disintegration/imaging"
"github.com/OpenListTeam/OpenList/v4/internal/model"
)
// BookLocalScraper 书籍本地刮削器
// 尝试从书籍文件中提取封面图片并以 base64 存入 Cover 字段,或保存到本地文件。
// - EPUB:解压 zip,提取 OPF 声明的封面或 cover.jpg/cover.png
// - PDF:尝试提取内嵌 JPEG 图片流作为封面
// - 其他/失败:降级为文件路径
type BookLocalScraper struct {
// ThumbnailMode 缩略图存储方式:"base64"(默认,存入数据库)或 "local"(存到本地文件)
ThumbnailMode string
// ThumbnailPath 本地存储路径,ThumbnailMode 为 "local" 时有效,默认 "/.thumbnail"
ThumbnailPath string
}
// NewBookLocalScraper 创建书籍本地刮削器
func NewBookLocalScraper() *BookLocalScraper {
return &BookLocalScraper{
ThumbnailMode: "base64",
ThumbnailPath: "/.thumbnail",
}
}
// NewBookLocalScraperWithConfig 创建带完整配置的书籍本地刮削器
func NewBookLocalScraperWithConfig(thumbnailMode, thumbnailPath string) *BookLocalScraper {
if thumbnailMode == "" {
thumbnailMode = "base64"
}
if thumbnailPath == "" {
thumbnailPath = "/.thumbnail"
}
return &BookLocalScraper{
ThumbnailMode: thumbnailMode,
ThumbnailPath: thumbnailPath,
}
}
// ScrapeBookLocal 从书籍文件流中提取封面图片,填充基本信息。
// reader 为书籍文件内容流,不能为 nil。
// 返回提取到的封面 base64 字符串(data URI),提取失败返回空字符串。
// 注意:本函数不会将文件路径作为 cover 降级,cover 为空表示提取失败。
func (s *BookLocalScraper) ScrapeBookLocal(item *model.MediaItem, reader io.Reader) error {
ext := strings.ToLower(getFileExt(item.FileName))
// 根据文件类型补充 MimeType
if item.MimeType == "" {
item.MimeType = mimeTypeByBookExt(ext)
}
// 去掉扩展名作为展示名
if item.ScrapedName == "" {
item.ScrapedName = trimExt(item.FileName)
}
// 从文件流提取封面,提取失败则 cover 保持为空
if reader != nil {
data, err := io.ReadAll(reader)
if err == nil && len(data) > 0 {
var coverData []byte
switch ext {
case ".epub":
coverData = extractEpubCoverData(data)
case ".pdf":
coverData = extractPDFCoverData(data)
}
if len(coverData) > 0 {
cover := s.storeCover(item.FolderPath, coverData)
if cover != "" {
item.Cover = cover
}
}
}
}
now := time.Now()
item.ScrapedAt = &now
return nil
}
// ExtractLocalCover 从书籍文件流中提取封面并根据配置存储(供外部调用)。
// 提取失败返回空字符串,不降级为文件路径。
func (s *BookLocalScraper) ExtractLocalCover(fileName string, filePath string, reader io.Reader) string {
if reader == nil {
return ""
}
ext := strings.ToLower(getFileExt(fileName))
data, err := io.ReadAll(reader)
if err != nil || len(data) == 0 {
return ""
}
var coverData []byte
switch ext {
case ".epub":
coverData = extractEpubCoverData(data)
case ".pdf":
coverData = extractPDFCoverData(data)
}
if len(coverData) == 0 {
return ""
}
return s.storeCover(filePath, coverData)
}
// storeCover 根据 ThumbnailMode 存储封面数据,返回 Cover 字段值
func (s *BookLocalScraper) storeCover(filePath string, coverData []byte) string {
if s.ThumbnailMode == "local" {
return s.saveCoverLocal(filePath, coverData)
}
// 默认 base64 模式
return imageDataToBase64Thumb(coverData)
}
// saveCoverLocal 将封面图片保存到本地文件系统,返回可访问的路径
func (s *BookLocalScraper) saveCoverLocal(filePath string, coverData []byte) string {
// 生成缩略图数据
img, _, err := image.Decode(bytes.NewReader(coverData))
if err != nil {
// 解码失败时直接保存原始数据
return s.writeLocalFile(filePath, coverData)
}
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
var buf bytes.Buffer
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil {
return ""
}
return s.writeLocalFile(filePath, buf.Bytes())
}
// writeLocalFile 将数据写入本地文件,返回内部访问路径
func (s *BookLocalScraper) writeLocalFile(filePath string, data []byte) string {
safeFileName := strings.ReplaceAll(strings.TrimPrefix(filePath, "/"), "/", "_") + ".jpg"
localDir := s.ThumbnailPath
if !filepath.IsAbs(localDir) {
if wd, err := os.Getwd(); err == nil {
localDir = filepath.Join(wd, localDir)
}
}
if err := os.MkdirAll(localDir, 0755); err != nil {
return ""
}
localFilePath := filepath.Join(localDir, safeFileName)
if err := os.WriteFile(localFilePath, data, 0644); err != nil {
return ""
}
thumbVFSPath := strings.TrimSuffix(s.ThumbnailPath, "/") + "/" + safeFileName
return buildInternalDownloadPath(thumbVFSPath)
}
// ── EPUB 封面提取 ─────────────────────────────────────────────────────────────
// epubContainer 解析 META-INF/container.xml
type epubContainer struct {
Rootfiles []struct {
FullPath string `xml:"full-path,attr"`
} `xml:"rootfiles>rootfile"`
}
// epubOPF 解析 OPF 文件,找封面 item
type epubOPF struct {
Metadata struct {
Metas []struct {
Name string `xml:"name,attr"`
Content string `xml:"content,attr"`
} `xml:"meta"`
} `xml:"metadata"`
Manifest struct {
Items []struct {
ID string `xml:"id,attr"`
Href string `xml:"href,attr"`
MediaType string `xml:"media-type,attr"`
} `xml:"item"`
} `xml:"manifest"`
}
// extractEpubCoverData 从 EPUB(zip)数据中提取封面图片原始字节
func extractEpubCoverData(data []byte) []byte {
r, err := zip.NewReader(bytes.NewReader(data), int64(len(data)))
if err != nil {
return nil
}
// 建立文件名→ZipFile 映射(忽略大小写)
fileMap := make(map[string]*zip.File, len(r.File))
for _, f := range r.File {
fileMap[strings.ToLower(f.Name)] = f
}
// 1. 尝试读取 META-INF/container.xml 找 OPF 路径
opfPath := ""
if cf, ok := fileMap["meta-inf/container.xml"]; ok {
if rc, err := cf.Open(); err == nil {
var container epubContainer
if xml.NewDecoder(rc).Decode(&container) == nil && len(container.Rootfiles) > 0 {
opfPath = container.Rootfiles[0].FullPath
}
rc.Close()
}
}
// OPF 所在目录(用于拼接相对路径)
opfDir := ""
if idx := strings.LastIndex(opfPath, "/"); idx >= 0 {
opfDir = opfPath[:idx+1]
}
// 2. 从 OPF 中找封面 item ID
coverHref := ""
if opfPath != "" {
if of, ok := fileMap[strings.ToLower(opfPath)]; ok {
if rc, err := of.Open(); err == nil {
var opf epubOPF
if xml.NewDecoder(rc).Decode(&opf) == nil {
// 先找 <meta name="cover" content="..."/> 指向的 item id
coverItemID := ""
for _, m := range opf.Metadata.Metas {
if strings.EqualFold(m.Name, "cover") && m.Content != "" {
coverItemID = m.Content
break
}
}
// 在 manifest 中找对应 item
for _, item := range opf.Manifest.Items {
if coverItemID != "" && item.ID == coverItemID {
coverHref = opfDir + item.Href
break
}
// 如果没有指定 cover id,找第一个图片类型的 item
if coverItemID == "" && strings.Contains(strings.ToLower(item.MediaType), "image") {
// 优先选择 id 或 href 中包含 "cover" 的
if strings.Contains(strings.ToLower(item.ID), "cover") ||
strings.Contains(strings.ToLower(item.Href), "cover") {
coverHref = opfDir + item.Href
break
}
}
}
// 如果还没找到,取第一个图片
if coverHref == "" && coverItemID == "" {
for _, item := range opf.Manifest.Items {
if strings.Contains(strings.ToLower(item.MediaType), "image") {
coverHref = opfDir + item.Href
break
}
}
}
}
rc.Close()
}
}
}
// 3. 常见封面文件名候选(按优先级)
candidates := []string{}
if coverHref != "" {
candidates = append(candidates, strings.ToLower(coverHref))
}
candidates = append(candidates,
"cover.jpg", "cover.jpeg", "cover.png",
"images/cover.jpg", "images/cover.jpeg", "images/cover.png",
"oebps/cover.jpg", "oebps/images/cover.jpg",
"oebps/cover.jpeg", "oebps/images/cover.jpeg",
)
for _, name := range candidates {
if f, ok := fileMap[name]; ok {
if rc, err := f.Open(); err == nil {
imgData, readErr := io.ReadAll(rc)
rc.Close()
if readErr == nil && len(imgData) > 0 {
return imgData
}
}
}
}
return nil
}
// extractEpubCoverBase64 从 EPUB(zip)数据中提取封面图片,返回 data URI base64(向后兼容)
func extractEpubCoverBase64(data []byte) string {
coverData := extractEpubCoverData(data)
if len(coverData) == 0 {
return ""
}
return imageDataToBase64Thumb(coverData)
}
// ── PDF 封面提取 ──────────────────────────────────────────────────────────────
// extractPDFCoverData 从 PDF 数据中提取第一个内嵌 JPEG 图片流,返回原始字节
// PDF 内嵌图片通常以 JPEG 流存储(SOI marker: 0xFF 0xD8),直接扫描字节流提取
func extractPDFCoverData(data []byte) []byte {
// 扫描 JPEG SOI (0xFF 0xD8 0xFF) 标记
for i := 0; i < len(data)-2; i++ {
if data[i] == 0xFF && data[i+1] == 0xD8 && data[i+2] == 0xFF {
// 找对应的 EOI (0xFF 0xD9)
for j := i + 2; j < len(data)-1; j++ {
if data[j] == 0xFF && data[j+1] == 0xD9 {
jpegData := data[i : j+2]
// 验证是否为有效图片(至少 1KB,避免误匹配小缩略图)
if len(jpegData) > 1024 {
return jpegData
}
break
}
}
}
}
return nil
}
// extractPDFCoverBase64 从 PDF 数据中提取第一个内嵌 JPEG 图片流,返回 data URI base64(向后兼容)
func extractPDFCoverBase64(data []byte) string {
coverData := extractPDFCoverData(data)
if len(coverData) == 0 {
return ""
}
return imageDataToBase64Thumb(coverData)
}
// ── 公共辅助 ──────────────────────────────────────────────────────────────────
// imageDataToBase64Thumb 将图片字节解码并缩放为 300px 宽缩略图,返回 data URI base64
func imageDataToBase64Thumb(data []byte) string {
img, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
// 解码失败时直接 base64 原始数据(可能是 PNG 等)
mimeType := detectImageMime(data)
return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data)
}
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
var buf bytes.Buffer
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil {
return ""
}
return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes())
}
// detectImageMime 根据文件头检测图片 MIME 类型
func detectImageMime(data []byte) string {
if len(data) < 4 {
return "image/jpeg"
}
switch {
case data[0] == 0xFF && data[1] == 0xD8:
return "image/jpeg"
case data[0] == 0x89 && data[1] == 0x50 && data[2] == 0x4E && data[3] == 0x47:
return "image/png"
case data[0] == 0x47 && data[1] == 0x49 && data[2] == 0x46:
return "image/gif"
case data[0] == 0x52 && data[1] == 0x49 && data[2] == 0x46 && data[3] == 0x46:
return "image/webp"
default:
return "image/jpeg"
}
}
// getFileExt 获取文件扩展名(含点号,小写)
func getFileExt(fileName string) string {
if idx := strings.LastIndex(fileName, "."); idx >= 0 {
return fileName[idx:]
}
return ""
}
// mimeTypeByBookExt 根据书籍文件扩展名返回 MIME 类型
func mimeTypeByBookExt(ext string) string {
switch ext {
case ".pdf":
return "application/pdf"
case ".epub":
return "application/epub+zip"
case ".mobi":
return "application/x-mobipocket-ebook"
case ".azw3":
return "application/vnd.amazon.ebook"
case ".txt":
return "text/plain"
case ".djvu":
return "image/vnd.djvu"
case ".cbz":
return "application/vnd.comicbook+zip"
case ".cbr":
return "application/vnd.comicbook-rar"
default:
return "application/octet-stream"
}
}
-317
View File
@@ -1,317 +0,0 @@
package scraper
import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strings"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/model"
)
const discogsDefaultBaseURL = "https://api.discogs.com"
// DiscogsScraper Discogs音乐刮削器
type DiscogsScraper struct {
Token string
BaseURL string // 可自定义的 Discogs API 地址
client *http.Client
}
// NewDiscogsScraper 创建Discogs刮削器
//
// 用户只需填写「域名 + API 前缀」,例如:
// 1. 留空:使用默认 https://api.discogs.com
// 2. 官方:api.discogs.com -> https://api.discogs.com
// 3. 反代:example.edgeone.run/api/discogs -> https://example.edgeone.run/api/discogs
// 4. 反代:example.edgeone.run/api/discogs/ -> https://example.edgeone.run/api/discogs(自动去尾斜杠)
//
// 后续会基于 BaseURL 直接拼接 /database/search、/releases/{id} 等具体路径。
func NewDiscogsScraper(token string, baseURL string) *DiscogsScraper {
base := strings.TrimRight(strings.TrimSpace(baseURL), "/")
if base == "" {
base = discogsDefaultBaseURL
}
// 协议头容错:用户没填 http/https 时自动补 https://
if !strings.HasPrefix(base, "http://") && !strings.HasPrefix(base, "https://") {
base = "https://" + base
}
// 再次去掉末尾多余的 /,避免后续拼接出 //database/search 这类双斜杠
base = strings.TrimRight(base, "/")
return &DiscogsScraper{
Token: token,
BaseURL: base,
client: &http.Client{Timeout: 15 * time.Second},
}
}
type discogsSearchResult struct {
Results []struct {
ID int `json:"id"`
Title string `json:"title"`
Type string `json:"type"`
Year string `json:"year"`
Thumb string `json:"thumb"`
CoverImage string `json:"cover_image"`
Genre []string `json:"genre"`
Style []string `json:"style"`
ResourceURL string `json:"resource_url"`
} `json:"results"`
}
type discogsReleaseDetail struct {
ID int `json:"id"`
Title string `json:"title"`
Year int `json:"year"`
CoverImage string `json:"cover_image"`
Notes string `json:"notes"`
Artists []struct {
Name string `json:"name"`
} `json:"artists"`
Genres []string `json:"genres"`
Styles []string `json:"styles"`
Tracklist []struct {
Position string `json:"position"`
Title string `json:"title"`
Duration string `json:"duration"`
} `json:"tracklist"`
Community struct {
Rating struct {
Average float32 `json:"average"`
} `json:"rating"`
} `json:"community"`
}
// parseMusicFileName 从音乐文件名中提取搜索关键词列表
// 规则:去掉扩展名后,按" - "或"-"分割,返回各部分作为候选搜索词
// 例如:
//
// "周杰伦 - 七里香.mp3" -> ["周杰伦 - 七里香", "周杰伦", "七里香"]
// "Coldplay-Yellow.flac" -> ["Coldplay-Yellow", "Coldplay", "Yellow"]
// "七里香.mp3" -> ["七里香"]
func parseMusicFileName(fileName string) []string {
// 去掉扩展名
if idx := strings.LastIndex(fileName, "."); idx > 0 {
ext := strings.ToLower(fileName[idx:])
if len(ext) <= 5 {
fileName = fileName[:idx]
}
}
fileName = strings.TrimSpace(fileName)
if fileName == "" {
return nil
}
var queries []string
// 完整文件名作为第一候选
queries = append(queries, fileName)
// 按" - "(带空格)分割
parts := strings.SplitN(fileName, " - ", 2)
if len(parts) == 2 {
before := strings.TrimSpace(parts[0])
after := strings.TrimSpace(parts[1])
if before != "" {
queries = append(queries, before)
}
if after != "" {
queries = append(queries, after)
}
} else {
// 按"-"(不带空格)分割
parts = strings.SplitN(fileName, "-", 2)
if len(parts) == 2 {
before := strings.TrimSpace(parts[0])
after := strings.TrimSpace(parts[1])
if before != "" {
queries = append(queries, before)
}
if after != "" {
queries = append(queries, after)
}
}
}
return queries
}
// doDiscogsSearch 执行单次Discogs搜索
func (s *DiscogsScraper) doDiscogsSearch(query string) (*discogsSearchResult, error) {
searchURL := fmt.Sprintf("%s/database/search?q=%s&type=release&token=%s",
s.BaseURL, url.QueryEscape(query), s.Token)
req, _ := http.NewRequest("GET", searchURL, nil)
req.Header.Set("User-Agent", "OpenList/4.0 +https://github.com/OpenListTeam/OpenList")
resp, err := s.client.Do(req)
if err != nil {
return nil, fmt.Errorf("Discogs搜索请求失败: %w", err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
var result discogsSearchResult
if err := json.Unmarshal(body, &result); err != nil {
return nil, fmt.Errorf("Discogs搜索结果解析失败: %w", err)
}
return &result, nil
}
// buildMusicQuery 根据 MediaItem 已有字段拼接 Discogs 搜索词
// 优先级:专辑名 > ScrapedName > 文件名解析
// 若有歌手或年份信息,则附加到搜索词中以提高精准度
func buildMusicQuery(base string, item *model.MediaItem) string {
q := base
// 附加歌手
if item.AlbumArtist != "" {
q = item.AlbumArtist + " " + q
}
// 附加年份(取 ReleaseDate 前4位)
if len(item.ReleaseDate) >= 4 {
q = q + " " + item.ReleaseDate[:4]
}
return strings.TrimSpace(q)
}
// ScrapeMusic 刮削音乐/专辑信息
func (s *DiscogsScraper) ScrapeMusic(item *model.MediaItem) error {
if s.Token == "" {
return fmt.Errorf("Discogs Token 未配置")
}
// 构建候选搜索词列表
var queries []string
if item.AlbumName != "" {
queries = append(queries, buildMusicQuery(item.AlbumName, item))
// 同时保留不带附加信息的纯专辑名作为降级候选
if item.AlbumArtist != "" || len(item.ReleaseDate) >= 4 {
queries = append(queries, item.AlbumName)
}
} else if item.ScrapedName != "" {
queries = append(queries, buildMusicQuery(item.ScrapedName, item))
if item.AlbumArtist != "" || len(item.ReleaseDate) >= 4 {
queries = append(queries, item.ScrapedName)
}
} else {
// 从文件名解析,按"-"分割分别搜索
baseQueries := parseMusicFileName(item.FileName)
for _, bq := range baseQueries {
enhanced := buildMusicQuery(bq, item)
if enhanced != bq {
// 先放带附加信息的精准词,再放原始词作为降级
queries = append(queries, enhanced)
}
queries = append(queries, bq)
}
}
if len(queries) == 0 {
return fmt.Errorf("无法从文件名中提取有效搜索词")
}
// 依次尝试各候选词,找到结果即停止(模糊匹配降级)
var searchResult *discogsSearchResult
var lastQuery string
for _, q := range queries {
result, err := s.doDiscogsSearch(q)
if err != nil {
return err
}
lastQuery = q
if len(result.Results) > 0 {
searchResult = result
break
}
}
if searchResult == nil || len(searchResult.Results) == 0 {
return fmt.Errorf("Discogs未找到匹配结果: %s", lastQuery)
}
first := searchResult.Results[0]
// 获取详情
detailURL := fmt.Sprintf("%s/releases/%d?token=%s", s.BaseURL, first.ID, s.Token)
detailReq, _ := http.NewRequest("GET", detailURL, nil)
detailReq.Header.Set("User-Agent", "OpenList/4.0 +https://github.com/OpenListTeam/OpenList")
detailResp, err := s.client.Do(detailReq)
if err != nil {
return fmt.Errorf("Discogs详情请求失败: %w", err)
}
defer detailResp.Body.Close()
detailBody, _ := io.ReadAll(detailResp.Body)
var detail discogsReleaseDetail
if err := json.Unmarshal(detailBody, &detail); err != nil {
return fmt.Errorf("Discogs详情解析失败: %w", err)
}
// 填充字段:已有值的字段不覆盖,仅补充空缺
// Discogs 的 title 就是专辑名(不是 "Artist - Album" 格式)
if detail.Title != "" {
if item.AlbumName == "" {
item.AlbumName = detail.Title
}
if item.ScrapedName == "" {
item.ScrapedName = detail.Title
}
}
if item.ReleaseDate == "" {
if detail.Year > 0 {
item.ReleaseDate = fmt.Sprintf("%d-01-01", detail.Year)
} else if first.Year != "" {
item.ReleaseDate = first.Year + "-01-01"
}
}
if item.Rating == 0 {
item.Rating = detail.Community.Rating.Average
}
if item.Plot == "" {
item.Plot = detail.Notes
}
if item.ExternalID == "" {
item.ExternalID = fmt.Sprintf("discogs:%d", detail.ID)
}
// 封面(存储 URL,前端直接展示)
if item.Cover == "" {
if detail.CoverImage != "" {
item.Cover = detail.CoverImage
} else if first.CoverImage != "" {
item.Cover = first.CoverImage
} else if first.Thumb != "" {
item.Cover = first.Thumb
}
}
// 类型
if item.Genre == "" {
genres := append(detail.Genres, detail.Styles...)
item.Genre = strings.Join(genres, ",")
}
// 艺术家:仅在 Authors 和 AlbumArtist 均为空时才填充(ID3已有值则保留)
artists := make([]string, 0, len(detail.Artists))
for _, a := range detail.Artists {
artists = append(artists, a.Name)
}
if len(artists) > 0 {
if item.Authors == "" {
authorsJSON, _ := json.Marshal(artists)
item.Authors = string(authorsJSON)
}
if item.AlbumArtist == "" {
item.AlbumArtist = artists[0]
}
}
now := time.Now()
item.ScrapedAt = &now
return nil
}
-483
View File
@@ -1,483 +0,0 @@
package scraper
import (
"bytes"
"encoding/base64"
"encoding/json"
"fmt"
"image"
"image/jpeg"
"io"
"net/http"
"net/url"
"regexp"
"strings"
"time"
"github.com/disintegration/imaging"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"golang.org/x/net/html"
)
const (
doubanSearchURL = "https://www.douban.com/search"
doubanBookCat = "1001"
doubanBase = "https://book.douban.com/"
)
var (
doubanBookURLPattern = regexp.MustCompile(`.*/subject/(\d+)/?`)
doubanDatePattern = regexp.MustCompile(`(\d{4})-(\d+)`)
doubanDefaultHeaders = map[string]string{
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Referer": doubanBase,
}
)
// DoubanScraper 豆瓣书籍刮削器(移植自 NewDouban.py)
type DoubanScraper struct {
client *http.Client
// ThumbnailMode 缩略图存储方式:"base64"(默认,存入数据库)或 "local"(存到本地文件)
ThumbnailMode string
// ThumbnailPath 本地存储路径,ThumbnailMode 为 "local" 时有效,默认 "/.thumbnail"
ThumbnailPath string
}
// NewDoubanScraper 创建豆瓣刮削器
func NewDoubanScraper() *DoubanScraper {
return &DoubanScraper{
client: &http.Client{Timeout: 20 * time.Second},
ThumbnailMode: "base64",
ThumbnailPath: "/.thumbnail",
}
}
// NewDoubanScraperWithConfig 创建带完整配置的豆瓣刮削器
func NewDoubanScraperWithConfig(thumbnailMode, thumbnailPath string) *DoubanScraper {
if thumbnailMode == "" {
thumbnailMode = "base64"
}
if thumbnailPath == "" {
thumbnailPath = "/.thumbnail"
}
return &DoubanScraper{
client: &http.Client{Timeout: 20 * time.Second},
ThumbnailMode: thumbnailMode,
ThumbnailPath: thumbnailPath,
}
}
// ScrapeBook 刮削书籍信息
func (s *DoubanScraper) ScrapeBook(item *model.MediaItem) error {
query := item.ScrapedName
if query == "" {
query = strings.TrimSuffix(item.FileName, strings.ToLower(item.FileName[strings.LastIndex(item.FileName, "."):]))
}
// 搜索书籍URL列表
bookURLs, err := s.searchBookURLs(query)
if err != nil {
return fmt.Errorf("豆瓣搜索失败: %w", err)
}
if len(bookURLs) == 0 {
return fmt.Errorf("豆瓣未找到匹配书籍: %s", query)
}
// 取第一个结果
bookDetail, err := s.loadBookDetail(bookURLs[0])
if err != nil {
return fmt.Errorf("豆瓣获取书籍详情失败: %w", err)
}
// 填充字段(仅填空字段,已有值不覆盖)
if item.ScrapedName == "" && bookDetail.Title != "" {
item.ScrapedName = bookDetail.Title
}
if item.Plot == "" {
item.Plot = bookDetail.Description
}
// 仅在豆瓣返回了有效封面图片URL且当前 item 还没有封面时才下载/覆盖
if item.Cover == "" && bookDetail.Cover != "" {
if cover := s.downloadAndStoreCover(item.FolderPath, bookDetail.Cover); cover != "" {
item.Cover = cover
} else {
// 下载失败时保留原 URL(降级)
item.Cover = bookDetail.Cover
}
}
if item.Rating == 0 {
item.Rating = bookDetail.Rating
}
if item.ReleaseDate == "" {
item.ReleaseDate = bookDetail.PublishedDate
}
if item.Publisher == "" {
item.Publisher = bookDetail.Publisher
}
if item.ISBN == "" {
item.ISBN = bookDetail.ISBN
}
if item.ExternalID == "" {
item.ExternalID = "douban:" + bookDetail.ID
}
if item.Authors == "" && len(bookDetail.Authors) > 0 {
authorsJSON, _ := json.Marshal(bookDetail.Authors)
item.Authors = string(authorsJSON)
}
if item.Genre == "" && len(bookDetail.Tags) > 0 {
item.Genre = strings.Join(bookDetail.Tags, ",")
}
now := time.Now()
item.ScrapedAt = &now
return nil
}
type doubanBookDetail struct {
ID string
Title string
Authors []string
Publisher string
PublishedDate string
Cover string
Rating float32
Description string
Tags []string
ISBN string
}
// searchBookURLs 搜索豆瓣书籍URL列表
func (s *DoubanScraper) searchBookURLs(query string) ([]string, error) {
searchURL := fmt.Sprintf("%s?cat=%s&q=%s", doubanSearchURL, doubanBookCat, url.QueryEscape(query))
body, err := s.doGet(searchURL)
if err != nil {
return nil, err
}
doc, err := html.Parse(strings.NewReader(string(body)))
if err != nil {
return nil, err
}
var bookURLs []string
var traverse func(*html.Node)
traverse = func(n *html.Node) {
if n.Type == html.ElementNode && n.Data == "a" {
for _, attr := range n.Attr {
if attr.Key == "class" && attr.Val == "nbg" {
for _, a := range n.Attr {
if a.Key == "href" {
parsed := s.calcURL(a.Val)
if parsed != "" && len(bookURLs) < 5 {
bookURLs = append(bookURLs, parsed)
}
}
}
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
traverse(c)
}
}
traverse(doc)
return bookURLs, nil
}
// calcURL 解析豆瓣搜索结果中的真实URL
func (s *DoubanScraper) calcURL(href string) string {
parsed, err := url.Parse(href)
if err != nil {
return ""
}
query := parsed.Query()
rawURL := query.Get("url")
if rawURL == "" {
return ""
}
decoded, err := url.QueryUnescape(rawURL)
if err != nil {
return ""
}
if doubanBookURLPattern.MatchString(decoded) {
return decoded
}
return ""
}
// loadBookDetail 加载书籍详情页
func (s *DoubanScraper) loadBookDetail(bookURL string) (*doubanBookDetail, error) {
body, err := s.doGet(bookURL)
if err != nil {
return nil, err
}
return s.parseBookHTML(bookURL, string(body))
}
// parseBookHTML 解析书籍HTML(移植自 DoubanBookHtmlParser.parse_book)
func (s *DoubanScraper) parseBookHTML(bookURL, content string) (*doubanBookDetail, error) {
doc, err := html.Parse(strings.NewReader(content))
if err != nil {
return nil, err
}
detail := &doubanBookDetail{}
// 提取豆瓣ID
if m := doubanBookURLPattern.FindStringSubmatch(bookURL); len(m) > 1 {
detail.ID = m[1]
}
var traverse func(*html.Node)
traverse = func(n *html.Node) {
if n.Type == html.ElementNode {
switch n.Data {
case "span":
// 标题
if getAttr(n, "property") == "v:itemreviewed" {
detail.Title = getTextContent(n)
}
// 元数据字段
if getAttr(n, "class") == "pl" {
text := getTextContent(n)
tail := getNextText(n)
switch {
case strings.HasPrefix(text, "作者") || strings.HasPrefix(text, "译者"):
// 作者通过链接提取
collectAuthors(n, &detail.Authors)
case strings.HasPrefix(text, "出版社"):
detail.Publisher = strings.TrimSpace(tail)
case strings.HasPrefix(text, "出版年"):
detail.PublishedDate = parseDoubanDate(strings.TrimSpace(tail))
case strings.HasPrefix(text, "ISBN"):
detail.ISBN = strings.TrimSpace(tail)
case strings.HasPrefix(text, "副标题"):
if detail.Title != "" {
detail.Title += ":" + strings.TrimSpace(tail)
}
}
}
// 评分
if getAttr(n, "property") == "v:average" {
ratingStr := getTextContent(n)
var rating float32
fmt.Sscanf(ratingStr, "%f", &rating)
detail.Rating = rating / 2
}
case "img":
// 封面图片:<img class="nbg" src="..." />
if getAttr(n, "class") == "nbg" {
src := getAttr(n, "src")
if src != "" && !strings.HasSuffix(src, "update_image") {
detail.Cover = src
}
}
// 备用:#mainpic 下的 img
if getAttr(n, "id") == "mainpic" || (n.Parent != nil && getAttr(n.Parent, "id") == "mainpic") {
src := getAttr(n, "src")
if src != "" && detail.Cover == "" {
detail.Cover = src
}
}
case "a":
// 标签(原封面逻辑移除,改为从img获取)
cls := getAttr(n, "class")
if strings.Contains(cls, "tag") {
tag := getTextContent(n)
if tag != "" {
detail.Tags = append(detail.Tags, tag)
}
}
case "div":
// 简介
if getAttr(n, "id") == "link-report" {
detail.Description = extractIntroText(n)
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
traverse(c)
}
}
traverse(doc)
return detail, nil
}
// HTML辅助函数
func getAttr(n *html.Node, key string) string {
for _, attr := range n.Attr {
if attr.Key == key {
return attr.Val
}
}
return ""
}
func getTextContent(n *html.Node) string {
var sb strings.Builder
var walk func(*html.Node)
walk = func(node *html.Node) {
if node.Type == html.TextNode {
sb.WriteString(node.Data)
}
for c := node.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
}
walk(n)
return strings.TrimSpace(sb.String())
}
func getNextText(n *html.Node) string {
if n.NextSibling != nil {
if n.NextSibling.Type == html.TextNode {
return strings.TrimSpace(n.NextSibling.Data)
}
return getTextContent(n.NextSibling)
}
return ""
}
func collectAuthors(n *html.Node, authors *[]string) {
var walk func(*html.Node)
walk = func(node *html.Node) {
if node.Type == html.ElementNode && node.Data == "a" {
href := getAttr(node, "href")
if strings.Contains(href, "/author") || strings.Contains(href, "/search") {
name := getTextContent(node)
if name != "" {
*authors = append(*authors, name)
}
}
}
for c := node.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
}
// 从父节点开始找
if n.Parent != nil {
walk(n.Parent)
}
}
func extractIntroText(n *html.Node) string {
var sb strings.Builder
var walk func(*html.Node)
walk = func(node *html.Node) {
if node.Type == html.ElementNode && node.Data == "div" {
if getAttr(node, "class") == "intro" {
sb.WriteString(getTextContent(node))
return
}
}
for c := node.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
}
walk(n)
return strings.TrimSpace(sb.String())
}
func parseDoubanDate(dateStr string) string {
if dateStr == "" {
return ""
}
if m := doubanDatePattern.FindStringSubmatch(dateStr); len(m) > 2 {
return fmt.Sprintf("%s-%s-01", m[1], m[2])
}
return dateStr
}
// downloadAndStoreCover 下载封面图片并根据 ThumbnailMode 存储
// 失败时返回空字符串
func (s *DoubanScraper) downloadAndStoreCover(filePath, imgURL string) string {
data, mimeType := s.downloadImage(imgURL)
if len(data) == 0 {
return ""
}
if s.ThumbnailMode == "local" {
// 本地存储模式
localScraper := NewBookLocalScraperWithConfig(s.ThumbnailMode, s.ThumbnailPath)
return localScraper.saveCoverLocal(filePath, data)
}
// Base64 模式(默认)
img, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data)
}
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
var buf bytes.Buffer
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil {
return ""
}
return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes())
}
// downloadCoverAsBase64 下载封面图片并转为 data URI Base64 字符串(向后兼容)
// 失败时返回空字符串
func (s *DoubanScraper) downloadCoverAsBase64(imgURL string) string {
data, mimeType := s.downloadImage(imgURL)
if len(data) == 0 {
return ""
}
img, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
return "data:" + mimeType + ";base64," + base64.StdEncoding.EncodeToString(data)
}
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
var buf bytes.Buffer
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 80}); err != nil {
return ""
}
return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes())
}
// downloadImage 下载图片,返回原始字节和 MIME 类型
func (s *DoubanScraper) downloadImage(imgURL string) ([]byte, string) {
req, err := http.NewRequest("GET", imgURL, nil)
if err != nil {
return nil, ""
}
for k, v := range doubanDefaultHeaders {
req.Header.Set(k, v)
}
resp, err := s.client.Do(req)
if err != nil {
return nil, ""
}
defer resp.Body.Close()
if resp.StatusCode != 200 {
return nil, ""
}
data, err := io.ReadAll(resp.Body)
if err != nil || len(data) == 0 {
return nil, ""
}
mimeType := resp.Header.Get("Content-Type")
if mimeType == "" {
mimeType = "image/jpeg"
}
return data, mimeType
}
// doGet 发送GET请求
func (s *DoubanScraper) doGet(reqURL string) ([]byte, error) {
req, err := http.NewRequest("GET", reqURL, nil)
if err != nil {
return nil, err
}
for k, v := range doubanDefaultHeaders {
req.Header.Set(k, v)
}
resp, err := s.client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode != 200 && resp.StatusCode != 201 {
return nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, reqURL)
}
return io.ReadAll(resp.Body)
}
-259
View File
@@ -1,259 +0,0 @@
package scraper
import (
"bytes"
"encoding/base64"
"fmt"
"image"
"image/jpeg"
"io"
"os"
"path/filepath"
"strings"
"time"
"github.com/disintegration/imaging"
"github.com/rwcarlsen/goexif/exif"
"github.com/OpenListTeam/OpenList/v4/internal/model"
)
// ImageScraper 图片刮削器
// 从图片 EXIF 中提取元数据,并生成缩略图写入 Cover 字段。
type ImageScraper struct {
// StoreThumbnail 为 true 时生成缩略图写入 Cover;
// 为 false 时直接用文件路径作为 Cover,节省数据库空间。
StoreThumbnail bool
// ThumbnailMode 缩略图存储方式:"base64"(默认,存入数据库)或 "local"(存到本地文件)
ThumbnailMode string
// ThumbnailPath 本地存储路径,ThumbnailMode 为 "local" 时有效,默认 "/.thumbnail"
ThumbnailPath string
}
// NewImageScraper 创建图片刮削器
// storeThumbnail 为 true 时生成缩略图写入 Cover。
func NewImageScraper(storeThumbnail bool) *ImageScraper {
return &ImageScraper{
StoreThumbnail: storeThumbnail,
ThumbnailMode: "base64",
ThumbnailPath: "/.thumbnail",
}
}
// NewImageScraperWithConfig 创建带完整配置的图片刮削器
func NewImageScraperWithConfig(storeThumbnail bool, thumbnailMode, thumbnailPath string) *ImageScraper {
if thumbnailMode == "" {
thumbnailMode = "base64"
}
if thumbnailPath == "" {
thumbnailPath = "/.thumbnail"
}
return &ImageScraper{
StoreThumbnail: storeThumbnail,
ThumbnailMode: thumbnailMode,
ThumbnailPath: thumbnailPath,
}
}
// ScrapeImage 刮削图片信息
// reader 为图片文件内容流,必须传入有效流才能生成缩略图。
// 当 reader 不为 nil 时:
// - 从 EXIF 读取拍摄时间、GPS 地点、相机型号+参数、评分
// - 生成 300px 宽缩略图,根据 ThumbnailMode 存储为 Base64 或本地文件
//
// 注意:绝不将文件路径作为 cover,reader 为 nil 或缩略图生成失败时 cover 保持为空。
func (s *ImageScraper) ScrapeImage(item *model.MediaItem, reader io.Reader) error {
// 去掉扩展名作为展示名(如果尚未设置)
if item.ScrapedName == "" {
item.ScrapedName = trimExt(item.FileName)
}
if reader == nil {
// 无文件流时无法生成缩略图,cover 保持为空,等待下次重试
now := time.Now()
item.ScrapedAt = &now
return nil
}
// 将流读入内存(EXIF 解析和图像解码都需要完整数据)
data, err := io.ReadAll(reader)
if err != nil || len(data) == 0 {
// 读取失败时 cover 保持为空
now := time.Now()
item.ScrapedAt = &now
return nil
}
// ── 1. 解析 EXIF ──────────────────────────────────────────────
s.parseEXIF(item, bytes.NewReader(data))
// ── 2. 生成缩略图并存储 ──────────────────────────────────────
if s.ThumbnailMode == "local" {
// 本地存储模式:将缩略图保存到指定目录,Cover 存储本地路径
if localPath := s.saveThumbnailLocal(item.FolderPath, data); localPath != "" {
item.Cover = localPath
}
// 本地存储失败时 cover 保持为空,不降级为文件路径
} else {
// Base64 模式(默认):生成缩略图 Base64 写入 Cover
if thumb := s.generateThumbnail(data); thumb != "" {
item.Cover = thumb
}
// 缩略图生成失败时 cover 保持为空,不降级为文件路径
}
now := time.Now()
item.ScrapedAt = &now
return nil
}
// saveThumbnailLocal 将缩略图保存到本地文件系统,返回可访问的路径
// 文件保存在 ThumbnailPath 目录下,文件名为原文件路径的 hash + .jpg
func (s *ImageScraper) saveThumbnailLocal(filePath string, data []byte) string {
// 生成缩略图数据
img, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
return ""
}
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
var buf bytes.Buffer
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 75}); err != nil {
return ""
}
// 构建本地存储路径:ThumbnailPath/原文件路径.jpg
// 将文件路径中的 / 替换为 _ 避免目录嵌套问题
safeFileName := strings.ReplaceAll(strings.TrimPrefix(filePath, "/"), "/", "_") + ".jpg"
localDir := s.ThumbnailPath
if !filepath.IsAbs(localDir) {
// 相对路径转为绝对路径(相对于工作目录)
if wd, err := os.Getwd(); err == nil {
localDir = filepath.Join(wd, localDir)
}
}
// 确保目录存在
if err := os.MkdirAll(localDir, 0755); err != nil {
return ""
}
localFilePath := filepath.Join(localDir, safeFileName)
if err := os.WriteFile(localFilePath, buf.Bytes(), 0644); err != nil {
return ""
}
// 返回内部访问路径(通过 /d/ 前缀访问)
// 缩略图路径格式:ThumbnailPath/safeFileName
thumbVFSPath := strings.TrimSuffix(s.ThumbnailPath, "/") + "/" + safeFileName
return buildInternalDownloadPath(thumbVFSPath)
}
// parseEXIF 从图片数据中解析 EXIF 元数据并写入 item
func (s *ImageScraper) parseEXIF(item *model.MediaItem, r io.Reader) {
x, err := exif.Decode(r)
if err != nil {
return
}
// ── 拍摄时间 → ReleaseDate ────────────────────────────────────
if item.ReleaseDate == "" {
if t, err := x.DateTime(); err == nil {
item.ReleaseDate = t.Format("2006-01-02")
}
}
// ── GPS 地点 → Authors(借用作者字段存储地点信息)────────────
if item.Authors == "" {
if lat, lon, err := x.LatLong(); err == nil {
item.Authors = fmt.Sprintf(`["GPS: %.6f, %.6f"]`, lat, lon)
}
}
// ── 相机型号 + 参数 → Genre ───────────────────────────────────
if item.Genre == "" {
var parts []string
if make, err := x.Get(exif.Make); err == nil {
if v, err := make.StringVal(); err == nil && v != "" {
parts = append(parts, strings.TrimSpace(v))
}
}
if model_, err := x.Get(exif.Model); err == nil {
if v, err := model_.StringVal(); err == nil && v != "" {
parts = append(parts, strings.TrimSpace(v))
}
}
if fnum, err := x.Get(exif.FNumber); err == nil {
if num, den, err := fnum.Rat2(0); err == nil && den != 0 {
parts = append(parts, fmt.Sprintf("f/%.1f", float64(num)/float64(den)))
}
}
if exp, err := x.Get(exif.ExposureTime); err == nil {
if num, den, err := exp.Rat2(0); err == nil && den != 0 {
if num == 1 {
parts = append(parts, fmt.Sprintf("1/%ds", den))
} else {
parts = append(parts, fmt.Sprintf("%d/%ds", num, den))
}
}
}
if iso, err := x.Get(exif.ISOSpeedRatings); err == nil {
if v, err := iso.Int(0); err == nil {
parts = append(parts, fmt.Sprintf("ISO%d", v))
}
}
if focal, err := x.Get(exif.FocalLength); err == nil {
if num, den, err := focal.Rat2(0); err == nil && den != 0 {
parts = append(parts, fmt.Sprintf("%.0fmm", float64(num)/float64(den)))
}
}
if len(parts) > 0 {
item.Genre = strings.Join(parts, ",")
}
}
// ── EXIF Rating(0-5 星)→ Rating(0-10 分)──────────────────
// Windows 资源管理器、Lightroom、Darktable 等软件会写入此标签
if item.Rating == 0 {
if ratingTag, err := x.Get(exif.FieldName("Rating")); err == nil {
if v, err := ratingTag.Int(0); err == nil && v > 0 {
// 1-5 星映射到 2-10 分
item.Rating = float32(v) * 2
}
}
}
}
// generateThumbnail 将图片数据缩放为 300px 宽缩略图,返回 data URI Base64 字符串
func (s *ImageScraper) generateThumbnail(data []byte) string {
img, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
return ""
}
// 缩放到宽度 300px,高度等比
thumb := imaging.Resize(img, 300, 0, imaging.Lanczos)
var buf bytes.Buffer
if err := jpeg.Encode(&buf, thumb, &jpeg.Options{Quality: 75}); err != nil {
return ""
}
return "data:image/jpeg;base64," + base64.StdEncoding.EncodeToString(buf.Bytes())
}
// buildInternalDownloadPath 将 VFS 文件路径转为内部代理下载路径
// 格式:/d/path/to/file(前端通过此路径访问文件内容)
func buildInternalDownloadPath(filePath string) string {
if !strings.HasPrefix(filePath, "/") {
filePath = "/" + filePath
}
return "/d" + filePath
}
// trimExt 去掉文件名的扩展名
func trimExt(fileName string) string {
if idx := strings.LastIndex(fileName, "."); idx > 0 {
return fileName[:idx]
}
return fileName
}
File diff suppressed because it is too large Load Diff
-113
View File
@@ -1,113 +0,0 @@
package model
import (
"time"
"gorm.io/gorm"
)
// MediaType 媒体类型,使用字符串便于后期扩展
type MediaType string
const (
MediaTypeVideo MediaType = "video"
MediaTypeMusic MediaType = "music"
MediaTypeImage MediaType = "image"
MediaTypeBook MediaType = "book"
)
// MediaScanPath 媒体库扫描路径(每个媒体库可配置多个扫描路径)
type MediaScanPath struct {
gorm.Model
ID uint `gorm:"primarykey" json:"id"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
DeletedAt gorm.DeletedAt `gorm:"index" json:"-"`
MediaType MediaType `gorm:"index;not null" json:"media_type"` // 所属媒体类型
Name string `json:"name"` // 路径名称(用于前端筛选显示)
Path string `gorm:"not null" json:"path"` // VFS 扫描路径
PathMerge bool `gorm:"default:false" json:"path_merge"` // 路径合并模式(子文件夹作为一个条目)
TypeTag string `json:"type_tag"` // 类型标签:电影、电视剧等
ContentTags string `json:"content_tags"` // 内容标签:喜剧、惊悚等(逗号分隔)
EnableScrape bool `gorm:"default:true" json:"enable_scrape"` // 是否启用刮削
LastScanAt *time.Time `json:"last_scan_at"` // 最后扫描时间
}
// MediaItem 媒体条目(统一表,通过 media_type 区分类型,便于后期扩展新类型)
type MediaItem struct {
gorm.Model
// 覆盖 gorm.Model 的 ID 字段,使 JSON 序列化为小写 "id",与前端保持一致
ID uint `gorm:"primarykey" json:"id"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
DeletedAt gorm.DeletedAt `gorm:"index" json:"-"`
// 基础信息
MediaType MediaType `gorm:"index;not null" json:"media_type"`
ScanPathID uint `gorm:"index" json:"scan_path_id"` // 关联的扫描路径ID
FileName string `gorm:"uniqueIndex:idx_media_folder_file_album" json:"file_name"`
FileSize int64 `json:"file_size"`
MimeType string `json:"mime_type"`
Hidden bool `gorm:"default:false" json:"hidden"`
// 刮削/编辑信息
ScrapedName string `json:"scraped_name"`
Description string `gorm:"type:text" json:"description"`
Cover string `json:"cover"` // 封面URL或本地路径
ReleaseDate string `json:"release_date"` // 发布时间,格式 YYYY-MM-DD
Rating float32 `json:"rating"` // 评分 0-10
Genre string `json:"genre"` // 类型,逗号分隔,如 "动作,科幻"
Authors string `gorm:"type:text" json:"authors"` // 作者/演员,JSON数组字符串
Plot string `gorm:"type:text" json:"plot"` // 剧情/内容介绍
Reviews string `gorm:"type:text" json:"reviews"` // 用户评价,JSON数组字符串
// 外部ID(用于刮削关联)
ExternalID string `json:"external_id"` // TMDB ID / Discogs ID / 豆瓣ID
// 音乐专属字段
AlbumName string `gorm:"uniqueIndex:idx_media_folder_file_album" json:"album_name"` // 所属专辑名
AlbumArtist string `json:"album_artist"` // 专辑艺术家
TrackNumber int `json:"track_number"` // 曲目编号
Duration int `json:"duration"` // 时长(秒)
Lyrics string `gorm:"type:text" json:"lyrics"` // LRC格式歌词
// 视频专属字段
VideoType string `json:"video_type"` // "movie" 或 "tv"
Season int `json:"season"` // 季(电视剧)
Episode int `json:"episode"` // 集(电视剧)
// 书籍专属字段
Publisher string `json:"publisher"` // 出版社
ISBN string `json:"isbn"` // ISBN
// 目录合并模式
IsFolder bool `gorm:"default:false" json:"is_folder"` // 是否为文件夹模式条目
FolderPath string `gorm:"index;uniqueIndex:idx_media_folder_file_album" json:"folder_path"` // 扫描根路径(恒定为扫描路径,与file_name+album_name组合唯一)
Episodes string `gorm:"type:text" json:"episodes"` // 选集信息,JSON数组,格式:[{file_name,index,title},...]
ScrapedAt *time.Time `json:"scraped_at"`
}
// MediaConfig 媒体库配置(每种类型一条记录)
type MediaConfig struct {
gorm.Model
ID uint `gorm:"primarykey" json:"id"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
DeletedAt gorm.DeletedAt `gorm:"index" json:"-"`
MediaType MediaType `gorm:"uniqueIndex;not null" json:"media_type"`
Enabled bool `gorm:"default:false" json:"enabled"`
LastScanAt *time.Time `json:"last_scan_at"`
LastScrapeAt *time.Time `json:"last_scrape_at"`
}
// MediaScanProgress 扫描进度(内存中维护,不持久化)
type MediaScanProgress struct {
MediaType MediaType `json:"media_type"`
ScanPathID uint `json:"scan_path_id,omitempty"` // 0 表示全部路径
Running bool `json:"running"`
Total int `json:"total"`
Done int `json:"done"`
Message string `json:"message"`
Error string `json:"error,omitempty"`
}
-2
View File
@@ -13,8 +13,6 @@ const (
S3
FTP
TRAFFIC
MEDIA
TRANSCODE
)
const (
+607
View File
@@ -0,0 +1,607 @@
package multipart
import (
"context"
"errors"
"fmt"
"io"
"math"
"os"
stdpath "path"
"path/filepath"
"sync"
"sync/atomic"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/conf"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
"github.com/google/uuid"
)
type State string
const (
StateReceiving State = "receiving"
StateCompleted State = "completed"
StateFailedRetriable State = "failed_retriable"
StateFailedPermanent State = "failed_permanent"
StateAborted State = "aborted"
)
// WindowSlots bounds the per-session disk footprint to WindowSlots*ChunkSize.
var WindowSlots = 8
const (
// defaultSessionTTL is the sliding inactivity timeout; it also serves as the
// grace period during which finished sessions remain queryable.
defaultSessionTTL = 30 * time.Minute
gcInterval = time.Minute
)
var (
ErrSessionNotFound = errors.New("multipart upload session not found")
ErrNotOwner = errors.New("multipart upload session belongs to another user")
// errAborted wraps context.Canceled so a driver blocked on the stream sees
// the abort as a canceled request and runs its cancellation cleanup.
errAborted = fmt.Errorf("multipart upload aborted: %w", context.Canceled)
)
// putFile is the pipeline tail: resolve the storage and run the regular upload
// path. It mirrors the checks of fs.putDirectly (internal/fs/put.go) but calls
// op.Put directly so the driver's progress callback can be observed.
// It is a variable so session tests can stub the storage layer out.
var putFile = func(ctx context.Context, dstDirPath string, fs *stream.FileStream, up driver.UpdateProgress) error {
storage, dstDirActualPath, err := op.GetStorageAndActualPath(dstDirPath)
if err != nil {
return err
}
if storage.Config().NoUpload {
return errs.UploadNotSupported
}
return op.Put(ctx, storage, dstDirActualPath, fs, up)
}
// Session is one multipart upload: metadata survives pipeline attempts, the
// window (chunk data) does not.
type Session struct {
ID string
Path string // full destination path (dir + name), already user-joined
DstDir string
Name string
Size int64
ChunkSize int64
Total int
Mimetype string
Modified time.Time
Hashes map[*utils.HashType]string
Creator *model.User
mu sync.Mutex
state State
err error
attempt int
win *Window
done chan struct{}
cancel context.CancelFunc
prevCRCs []uint32
prevSet []bool
storagePct atomic.Uint64 // math.Float64bits of the driver progress (0-100)
lastActive atomic.Int64 // unix nano
}
func (s *Session) touch() { s.lastActive.Store(time.Now().UnixNano()) }
func (s *Session) setStoragePct(p float64) { s.storagePct.Store(math.Float64bits(p)) }
// Snapshot is the wire representation of a session used by all endpoints.
type SessionSnapshot struct {
ID string `json:"upload_id"`
State State `json:"state"`
Attempt int `json:"attempt"`
Path string `json:"path"`
Size int64 `json:"size"`
ChunkSize int64 `json:"chunk_size"`
TotalChunks int `json:"total_chunks"`
Received [][2]int `json:"received"`
ReceivedBytes int64 `json:"received_bytes"`
Frontier int `json:"frontier"`
StorageProgress float64 `json:"storage_progress"`
Error string `json:"error,omitempty"`
}
func (s *Session) snapshotLocked() SessionSnapshot {
snap := SessionSnapshot{
ID: s.ID,
State: s.state,
Attempt: s.attempt,
Path: s.Path,
Size: s.Size,
ChunkSize: s.ChunkSize,
TotalChunks: s.Total,
Received: [][2]int{},
StorageProgress: math.Float64frombits(s.storagePct.Load()),
}
if s.err != nil {
snap.Error = s.err.Error()
}
switch {
case s.state == StateCompleted:
snap.Received = [][2]int{{0, s.Total - 1}}
snap.ReceivedBytes = s.Size
snap.Frontier = s.Total
snap.StorageProgress = 100
case s.win != nil:
ws := s.win.Snapshot()
snap.Received = ws.Received
snap.ReceivedBytes = ws.ReceivedBytes
snap.Frontier = ws.Frontier
}
return snap
}
func (s *Session) Snapshot() SessionSnapshot {
s.mu.Lock()
defer s.mu.Unlock()
return s.snapshotLocked()
}
// InitReq carries everything the handler parsed from the init request.
type InitReq struct {
User *model.User
Path string // full destination path, already user-joined
Size int64
ChunkSize int64 // final chunk size in bytes, already clamped by the handler
Mimetype string
Modified time.Time
Hashes map[*utils.HashType]string
}
// Manager owns all live sessions. Sessions are in-memory only (aligned with
// upload tasks not being persisted); a restart drops them and the ring files
// are swept on the next start.
type Manager struct {
mu sync.Mutex
byID map[string]*Session
byKey map[string]string
gcOnce sync.Once
ttl time.Duration // 0 means defaultSessionTTL; tests shrink it per instance
}
func (m *Manager) sessionTTL() time.Duration {
if m.ttl > 0 {
return m.ttl
}
return defaultSessionTTL
}
var DefaultManager = &Manager{
byID: make(map[string]*Session),
byKey: make(map[string]string),
}
func (m *Manager) dir() string {
return filepath.Join(conf.Conf.TempDir, "multipart")
}
func sessionKey(userID uint, path string, size int64) string {
return fmt.Sprintf("%d|%s|%d", userID, path, size)
}
// hashesQualifyResume reports whether two hash sets prove the client is
// re-uploading the same file: they must share at least one hash type and
// agree on every shared one. Path+size alone is NOT enough to resume into a
// receiving session — buffered chunks of a different same-sized file would be
// silently mixed into the result.
func hashesQualifyResume(old, new map[*utils.HashType]string) bool {
shared := false
for t, ov := range old {
if nv, ok := new[t]; ok {
if ov != nv {
return false
}
shared = true
}
}
return shared
}
// StartGC sweeps ring files orphaned by a previous run and starts the expiry
// loop. It is called at server startup so orphans are reclaimed even if no
// multipart upload ever happens again; Init also calls it, so embedders that
// skip the server wiring still get GC lazily.
func (m *Manager) StartGC() {
m.ensureGC()
}
func (m *Manager) ensureGC() {
m.gcOnce.Do(func() {
// sweep ring files orphaned by a previous run; bootstrap's CleanTempDir
// only runs when no transfer tasks are pending, so do not rely on it
_ = os.RemoveAll(m.dir())
go func() {
ticker := time.NewTicker(gcInterval)
for range ticker.C {
m.gc()
}
}()
})
}
func (m *Manager) gc() {
deadline := time.Now().Add(-m.sessionTTL()).UnixNano()
m.mu.Lock()
var expired []*Session
for _, s := range m.byID {
if s.lastActive.Load() < deadline {
expired = append(expired, s)
}
}
m.mu.Unlock()
for _, s := range expired {
m.terminate(s, errors.New("multipart upload session expired"))
}
}
// terminate aborts a session (if still receiving) and drops it from the maps.
func (m *Manager) terminate(s *Session, cause error) {
s.mu.Lock()
if s.state == StateReceiving {
s.state = StateAborted
s.err = cause
}
s.killAttemptLocked()
s.mu.Unlock()
m.remove(s)
}
// killAttemptLocked stops the running pipeline attempt: the context cancel
// interrupts drivers blocked on network I/O, and closing the window wakes a
// driver blocked in Read (context cancellation cannot interrupt cond.Wait).
// The caller must hold s.mu and must have set the final state first.
func (s *Session) killAttemptLocked() {
if s.cancel != nil {
s.cancel()
}
if s.win != nil {
_ = s.win.CloseWithError(errAborted)
}
}
func (m *Manager) remove(s *Session) {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.byID, s.ID)
key := sessionKey(s.Creator.ID, s.Path, s.Size)
if m.byKey[key] == s.ID {
delete(m.byKey, key)
}
}
func (m *Manager) get(user *model.User, id string) (*Session, error) {
m.mu.Lock()
s, ok := m.byID[id]
m.mu.Unlock()
if !ok {
return nil, ErrSessionNotFound
}
if s.Creator.ID != user.ID {
return nil, ErrNotOwner
}
return s, nil
}
// Init creates a session and starts its pipeline, or returns the live session
// for the same (user, path, size) so an interrupted client resumes implicitly.
func (m *Manager) Init(req InitReq) (SessionSnapshot, bool, error) {
m.ensureGC()
if req.Size <= 0 {
return SessionSnapshot{}, false, fmt.Errorf("multipart upload requires a positive X-File-Size, got %d", req.Size)
}
if req.ChunkSize <= 0 {
return SessionSnapshot{}, false, fmt.Errorf("invalid chunk size %d", req.ChunkSize)
}
key := sessionKey(req.User.ID, req.Path, req.Size)
m.mu.Lock()
if id, ok := m.byKey[key]; ok {
if s, ok := m.byID[id]; ok {
s.mu.Lock()
st := s.state
s.mu.Unlock()
// failed_retriable resumes unconditionally: nothing of the old
// attempt survives except CRCs, and the re-fill CRC check catches
// a changed file. A receiving session still holds data, so it only
// resumes when hashes prove it is the same file.
if st == StateFailedRetriable ||
(st == StateReceiving && hashesQualifyResume(s.Hashes, req.Hashes)) {
m.mu.Unlock()
s.touch()
return s.Snapshot(), true, nil
}
// finished session, or same path+size without proof of identity:
// drop the old session and start fresh
m.mu.Unlock()
m.terminate(s, errors.New("superseded by a new upload of the same path and size"))
m.mu.Lock()
}
}
m.mu.Unlock()
dstDir, name := stdpath.Split(req.Path)
s := &Session{
ID: uuid.NewString(),
Path: req.Path,
DstDir: dstDir,
Name: name,
Size: req.Size,
ChunkSize: req.ChunkSize,
Mimetype: req.Mimetype,
Modified: req.Modified,
Hashes: req.Hashes,
Creator: req.User,
state: StateReceiving,
}
s.touch()
s.mu.Lock()
if err := m.startAttemptLocked(s); err != nil {
s.mu.Unlock()
return SessionSnapshot{}, false, err
}
s.Total = s.win.TotalChunks()
snap := s.snapshotLocked()
s.mu.Unlock()
m.mu.Lock()
m.byID[s.ID] = s
m.byKey[key] = s.ID
m.mu.Unlock()
return snap, false, nil
}
// startAttemptLocked builds a fresh window and spawns the pipeline goroutine.
// The caller must hold s.mu.
func (m *Manager) startAttemptLocked(s *Session) error {
win, err := NewWindow(m.dir(), fmt.Sprintf("%s.%d", s.ID, s.attempt), s.ChunkSize, s.Size, WindowSlots)
if err != nil {
return err
}
ctx, cancel := context.WithCancel(context.WithValue(context.Background(), conf.UserKey, s.Creator))
done := make(chan struct{})
s.win = win
s.cancel = cancel
s.done = done
s.state = StateReceiving
s.err = nil
s.setStoragePct(0)
fileStream := &stream.FileStream{
Obj: &model.Object{
Name: s.Name,
Size: s.Size,
Modified: s.Modified,
HashInfo: utils.NewHashInfoByMap(s.Hashes),
},
Reader: win,
Mimetype: s.Mimetype,
}
fileStream.Add(win)
dstDir := s.DstDir
put := putFile // capture: the seam must not be read after spawn
go func() {
err := put(ctx, dstDir, fileStream, s.setStoragePct)
s.finishAttempt(win, err)
close(done)
}()
return nil
}
// finishAttempt records the pipeline outcome and harvests the CRC table for
// re-fill verification. The window's data is gone at this point (op.Put closed
// it); only metadata survives.
func (s *Session) finishAttempt(win *Window, err error) {
crcs, set := win.CRCs()
_ = win.Close() // op.Put already closed it; make sure the ring file is gone anyway
s.mu.Lock()
defer s.mu.Unlock()
s.touch()
if s.win == win {
s.win = nil
}
s.prevCRCs, s.prevSet = crcs, set
switch {
case err == nil:
s.state = StateCompleted
s.err = nil
s.setStoragePct(100)
case s.state != StateReceiving:
// Abort/expiry already labeled this attempt; keep that state.
if s.err == nil {
s.err = err
}
case isPermanentPutError(err):
s.state = StateFailedPermanent
s.err = err
default:
s.state = StateFailedRetriable
s.err = err
s.attempt++
}
}
func isPermanentPutError(err error) bool {
if errors.Is(err, context.Canceled) {
return false
}
for _, target := range []error{
errs.UploadNotSupported,
errs.PermissionDenied,
errs.StorageNotFound,
errs.ObjectAlreadyExists,
errs.RelativePath,
errs.IgnoredSystemFile,
} {
if errors.Is(err, target) {
return true
}
}
return false
}
// Chunk feeds one chunk into the session. Re-sending chunk 0 to a
// failed_retriable session re-fills it: a fresh window and pipeline attempt.
func (m *Manager) Chunk(user *model.User, id string, idx int, body io.Reader) (SessionSnapshot, error) {
s, err := m.get(user, id)
if err != nil {
return SessionSnapshot{}, err
}
s.touch()
s.mu.Lock()
switch s.state {
case StateReceiving:
case StateFailedRetriable:
if idx != 0 {
snap := s.snapshotLocked()
s.mu.Unlock()
return snap, fmt.Errorf("upload attempt failed, resend from chunk 0 to retry: %w", s.err)
}
if err := m.startAttemptLocked(s); err != nil {
snap := s.snapshotLocked()
s.mu.Unlock()
return snap, err
}
case StateCompleted:
snap := s.snapshotLocked()
s.mu.Unlock()
return snap, nil // idempotent: stragglers after rapid-upload/finish succeed
default:
snap := s.snapshotLocked()
s.mu.Unlock()
return snap, fmt.Errorf("session is %s: %w", s.state, s.err)
}
win := s.win
var prevCRC uint32
hasPrev := false
if idx < len(s.prevSet) && s.prevSet[idx] {
prevCRC, hasPrev = s.prevCRCs[idx], true
}
s.mu.Unlock()
crc, err := win.WriteChunk(idx, body)
if err != nil {
// A closed window means the pipeline ended while this chunk was in
// flight — rapid upload makes this the NORMAL case: the driver
// succeeds off the hash alone with chunks still arriving. The window
// closes (op.Put's defer) moments before the verdict is recorded, so
// wait for the verdict instead of racing it, then absorb the chunk
// idempotently if the upload in fact succeeded.
if errors.Is(err, ErrClosed) || errors.Is(err, context.Canceled) {
s.mu.Lock()
done := s.done
s.mu.Unlock()
select {
case <-done:
case <-time.After(10 * time.Second): // pipeline teardown is µs-scale; never expected
}
}
s.mu.Lock()
completed := s.state == StateCompleted
snap := s.snapshotLocked()
s.mu.Unlock()
if completed {
return snap, nil
}
return snap, err
}
if hasPrev && crc != prevCRC {
err := fmt.Errorf("chunk %d content changed between attempts, aborting", idx)
s.mu.Lock()
s.state = StateFailedPermanent
s.err = err
s.killAttemptLocked()
s.mu.Unlock()
return s.Snapshot(), err
}
return s.Snapshot(), nil
}
// Complete waits for the pipeline outcome. It refuses to block while chunks
// are still missing, so a buggy client cannot park a connection for the TTL.
func (m *Manager) Complete(ctx context.Context, user *model.User, id string) (SessionSnapshot, error) {
s, err := m.get(user, id)
if err != nil {
return SessionSnapshot{}, err
}
s.touch()
for {
s.mu.Lock()
st := s.state
done := s.done
if st == StateReceiving && s.win != nil {
if ws := s.win.Snapshot(); ws.ReceivedBytes < s.Size {
snap := s.snapshotLocked()
s.mu.Unlock()
return snap, fmt.Errorf("cannot complete: %d of %d bytes received", ws.ReceivedBytes, s.Size)
}
}
s.mu.Unlock()
if st != StateReceiving {
break
}
select {
case <-done:
case <-ctx.Done():
return s.Snapshot(), ctx.Err()
}
}
s.touch()
s.mu.Lock()
st, serr := s.state, s.err
snap := s.snapshotLocked()
s.mu.Unlock()
if st == StateCompleted {
m.remove(s) // served its purpose; frees the key for future uploads
return snap, nil
}
return snap, fmt.Errorf("upload failed (%s): %w", st, serr)
}
// Status looks a session up by id.
func (m *Manager) Status(user *model.User, id string) (SessionSnapshot, error) {
s, err := m.get(user, id)
if err != nil {
return SessionSnapshot{}, err
}
s.touch()
return s.Snapshot(), nil
}
// Find looks a live session up by destination path and size, for resume discovery.
func (m *Manager) Find(user *model.User, path string, size int64) (SessionSnapshot, error) {
m.mu.Lock()
id, ok := m.byKey[sessionKey(user.ID, path, size)]
m.mu.Unlock()
if !ok {
return SessionSnapshot{}, ErrSessionNotFound
}
return m.Status(user, id)
}
// Abort cancels the pipeline and forgets the session immediately.
func (m *Manager) Abort(user *model.User, id string) error {
s, err := m.get(user, id)
if err != nil {
return err
}
m.terminate(s, errors.New("multipart upload aborted by client"))
return nil
}
+650
View File
@@ -0,0 +1,650 @@
package multipart
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"os"
"sync"
"testing"
"time"
"github.com/OpenListTeam/OpenList/v4/internal/conf"
"github.com/OpenListTeam/OpenList/v4/internal/driver"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/stream"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
)
// Tests in this file share the putFile seam and must not run in parallel.
func setupSessionTest(t *testing.T) *Manager {
t.Helper()
oldConf := conf.Conf
conf.Conf = conf.DefaultConfig(t.TempDir())
t.Cleanup(func() { conf.Conf = oldConf })
return &Manager{byID: make(map[string]*Session), byKey: make(map[string]string)}
}
func stubPut(t *testing.T, fn func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error) {
t.Helper()
orig := putFile
putFile = fn
t.Cleanup(func() { putFile = orig })
}
func testUser() *model.User { return &model.User{ID: 7, Username: "tester"} }
func initReq(user *model.User, size, chunkSize int64) InitReq {
return InitReq{
User: user,
Path: "/local/test.bin",
Size: size,
ChunkSize: chunkSize,
Mimetype: "application/octet-stream",
Modified: time.Unix(1700000000, 0),
}
}
func sendChunk(t *testing.T, m *Manager, user *model.User, id string, data []byte, idx int, chunkSize int64) SessionSnapshot {
t.Helper()
snap, err := m.Chunk(user, id, idx, bytes.NewReader(chunkOf(data, idx, chunkSize)))
if err != nil {
t.Fatalf("Chunk(%d): %v", idx, err)
}
return snap
}
func waitState(t *testing.T, m *Manager, user *model.User, id string, want State) SessionSnapshot {
t.Helper()
deadline := time.Now().Add(10 * time.Second)
for {
snap, err := m.Status(user, id)
if err != nil {
t.Fatalf("Status: %v", err)
}
if snap.State == want {
return snap
}
if time.Now().After(deadline) {
t.Fatalf("session state = %s, want %s (err: %s)", snap.State, want, snap.Error)
}
time.Sleep(2 * time.Millisecond)
}
}
func TestSessionHappyPath(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(3*chunkSize + 300)
data := genData(totalSize)
user := testUser()
got := make(chan []byte, 1)
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
if dst != "/local/" {
return fmt.Errorf("unexpected dst dir %q", dst)
}
if fs.GetName() != "test.bin" || fs.GetSize() != totalSize {
return fmt.Errorf("unexpected stream meta %s/%d", fs.GetName(), fs.GetSize())
}
b, err := io.ReadAll(fs)
if err != nil {
return err
}
up(100)
got <- b
return nil
})
snap, resumed, err := m.Init(initReq(user, totalSize, chunkSize))
if err != nil || resumed {
t.Fatalf("Init = (resumed=%v, err=%v)", resumed, err)
}
if snap.TotalChunks != 4 || snap.State != StateReceiving {
t.Fatalf("init snapshot = %+v", snap)
}
for _, idx := range []int{1, 0, 3, 2} { // out of order on purpose
sendChunk(t, m, user, snap.ID, data, idx, chunkSize)
}
final, err := m.Complete(context.Background(), user, snap.ID)
if err != nil {
t.Fatalf("Complete: %v", err)
}
if final.State != StateCompleted || final.StorageProgress != 100 {
t.Fatalf("final snapshot = %+v", final)
}
if !bytes.Equal(<-got, data) {
t.Fatal("driver received different bytes")
}
if _, err := m.Status(user, snap.ID); !errors.Is(err, ErrSessionNotFound) {
t.Fatalf("session should be removed after Complete, got %v", err)
}
}
func TestSessionRetriableRefill(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(2 * chunkSize)
data := genData(totalSize)
user := testUser()
attempts := 0
got := make(chan []byte, 1)
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
attempts++
if attempts == 1 {
buf := make([]byte, chunkSize)
if _, err := io.ReadFull(fs, buf); err != nil {
return err
}
return errors.New("transient storage hiccup")
}
b, err := io.ReadAll(fs)
if err != nil {
return err
}
got <- b
return nil
})
snap, _, err := m.Init(initReq(user, totalSize, chunkSize))
if err != nil {
t.Fatalf("Init: %v", err)
}
sendChunk(t, m, user, snap.ID, data, 0, chunkSize)
failed := waitState(t, m, user, snap.ID, StateFailedRetriable)
if failed.Attempt != 1 {
t.Fatalf("attempt = %d, want 1", failed.Attempt)
}
if len(failed.Received) != 0 {
t.Fatalf("failed session must report nothing received, got %v", failed.Received)
}
// chunks other than 0 are rejected until the client restarts the fill
if _, err := m.Chunk(user, snap.ID, 1, bytes.NewReader(chunkOf(data, 1, chunkSize))); err == nil {
t.Fatal("chunk 1 on failed_retriable session: expected error")
}
// re-fill from chunk 0 respawns the pipeline
sendChunk(t, m, user, snap.ID, data, 0, chunkSize)
sendChunk(t, m, user, snap.ID, data, 1, chunkSize)
final, err := m.Complete(context.Background(), user, snap.ID)
if err != nil {
t.Fatalf("Complete after refill: %v", err)
}
if final.State != StateCompleted || attempts != 2 {
t.Fatalf("state=%s attempts=%d, want completed/2", final.State, attempts)
}
if !bytes.Equal(<-got, data) {
t.Fatal("driver received different bytes after refill")
}
}
func TestSessionPermanentFailure(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
data := genData(chunkSize)
user := testUser()
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
return fmt.Errorf("denied: %w", errs.PermissionDenied)
})
snap, _, err := m.Init(initReq(user, chunkSize, chunkSize))
if err != nil {
t.Fatalf("Init: %v", err)
}
waitState(t, m, user, snap.ID, StateFailedPermanent)
if _, err := m.Chunk(user, snap.ID, 0, bytes.NewReader(data)); err == nil {
t.Fatal("chunk on failed_permanent session: expected error")
}
if _, err := m.Complete(context.Background(), user, snap.ID); err == nil {
t.Fatal("Complete on failed_permanent session: expected error")
}
}
func TestRefillCRCMismatch(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(2 * chunkSize)
data := genData(totalSize)
user := testUser()
attempts := 0
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
attempts++
if attempts == 1 {
buf := make([]byte, chunkSize)
if _, err := io.ReadFull(fs, buf); err != nil {
return err
}
return errors.New("transient")
}
_, err := io.ReadAll(fs)
return err
})
snap, _, err := m.Init(initReq(user, totalSize, chunkSize))
if err != nil {
t.Fatalf("Init: %v", err)
}
sendChunk(t, m, user, snap.ID, data, 0, chunkSize)
waitState(t, m, user, snap.ID, StateFailedRetriable)
tampered := genData(chunkSize + 5)[:chunkSize] // different content, same length
if _, err := m.Chunk(user, snap.ID, 0, bytes.NewReader(tampered)); err == nil {
t.Fatal("re-fill with changed content: expected error")
}
waitState(t, m, user, snap.ID, StateFailedPermanent)
}
func TestCompleteRefusesIncomplete(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(3 * chunkSize)
data := genData(totalSize)
user := testUser()
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
_, err := io.ReadAll(fs)
return err
})
snap, _, err := m.Init(initReq(user, totalSize, chunkSize))
if err != nil {
t.Fatalf("Init: %v", err)
}
sendChunk(t, m, user, snap.ID, data, 0, chunkSize)
if _, err := m.Complete(context.Background(), user, snap.ID); err == nil {
t.Fatal("Complete with missing chunks: expected error")
}
// unblock the pipeline goroutine before the test tears down
if err := m.Abort(user, snap.ID); err != nil {
t.Fatalf("Abort: %v", err)
}
}
func TestAbortAndOwnership(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(4 * chunkSize)
data := genData(totalSize)
user := testUser()
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
_, err := io.ReadAll(fs)
return err
})
snap, _, err := m.Init(initReq(user, totalSize, chunkSize))
if err != nil {
t.Fatalf("Init: %v", err)
}
sendChunk(t, m, user, snap.ID, data, 0, chunkSize)
stranger := &model.User{ID: 99, Username: "stranger"}
if _, err := m.Status(stranger, snap.ID); !errors.Is(err, ErrNotOwner) {
t.Fatalf("stranger Status err = %v, want ErrNotOwner", err)
}
if err := m.Abort(stranger, snap.ID); !errors.Is(err, ErrNotOwner) {
t.Fatalf("stranger Abort err = %v, want ErrNotOwner", err)
}
if err := m.Abort(user, snap.ID); err != nil {
t.Fatalf("Abort: %v", err)
}
if _, err := m.Status(user, snap.ID); !errors.Is(err, ErrSessionNotFound) {
t.Fatalf("Status after Abort err = %v, want ErrSessionNotFound", err)
}
}
func TestExpiry(t *testing.T) {
m := setupSessionTest(t)
m.ttl = 30 * time.Millisecond
const chunkSize = 1024
user := testUser()
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
_, err := io.ReadAll(fs)
return err
})
snap, _, err := m.Init(initReq(user, 4*chunkSize, chunkSize))
if err != nil {
t.Fatalf("Init: %v", err)
}
time.Sleep(60 * time.Millisecond)
m.gc()
if _, err := m.Status(user, snap.ID); !errors.Is(err, ErrSessionNotFound) {
t.Fatalf("Status after expiry err = %v, want ErrSessionNotFound", err)
}
// the terminated pipeline must release the ring file
deadline := time.Now().Add(5 * time.Second)
for {
entries, _ := os.ReadDir(m.dir())
if len(entries) == 0 {
break
}
if time.Now().After(deadline) {
names := make([]string, 0, len(entries))
for _, e := range entries {
names = append(names, e.Name())
}
t.Fatalf("ring files not cleaned up after expiry: %v", names)
}
time.Sleep(5 * time.Millisecond)
}
}
func TestInitResume(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(4 * chunkSize)
data := genData(totalSize)
user := testUser()
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
_, err := io.ReadAll(fs)
return err
})
hashed := initReq(user, totalSize, chunkSize)
hashed.Hashes = map[*utils.HashType]string{utils.MD5: "0123456789abcdef0123456789abcdef"}
snap, _, err := m.Init(hashed)
if err != nil {
t.Fatalf("Init: %v", err)
}
sendChunk(t, m, user, snap.ID, data, 2, chunkSize)
// same hashes prove the same file: resume with buffered chunks skippable
again, resumed, err := m.Init(hashed)
if err != nil || !resumed {
t.Fatalf("hashed re-Init = (resumed=%v, err=%v), want resumed", resumed, err)
}
if again.ID != snap.ID {
t.Fatalf("resumed session id = %s, want %s", again.ID, snap.ID)
}
if len(again.Received) != 1 || again.Received[0] != [2]int{2, 2} {
t.Fatalf("resumed received = %v, want [[2,2]]", again.Received)
}
// a different size is a different upload
other, resumed, err := m.Init(initReq(user, totalSize+1, chunkSize))
if err != nil || resumed || other.ID == snap.ID {
t.Fatalf("different-size Init = (id=%s, resumed=%v, err=%v)", other.ID, resumed, err)
}
if _, err := m.Find(user, "/local/test.bin", totalSize); err != nil {
t.Fatalf("Find: %v", err)
}
if _, err := m.Find(user, "/local/nope.bin", totalSize); !errors.Is(err, ErrSessionNotFound) {
t.Fatalf("Find miss err = %v, want ErrSessionNotFound", err)
}
// without hashes, path+size cannot prove identity against a receiving
// session holding buffered data — the old session must be dropped, or a
// same-sized different file would be silently mixed into the result
bare := initReq(user, totalSize, chunkSize)
fresh, resumed, err := m.Init(bare)
if err != nil || resumed || fresh.ID == snap.ID {
t.Fatalf("bare re-Init = (id=%s, resumed=%v, err=%v), want a fresh session", fresh.ID, resumed, err)
}
if _, err := m.Status(user, snap.ID); !errors.Is(err, ErrSessionNotFound) {
t.Fatalf("superseded session err = %v, want ErrSessionNotFound", err)
}
}
func TestResumeFailedRetriableWithoutHash(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(2 * chunkSize)
data := genData(totalSize)
user := testUser()
attempts := 0
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
attempts++
if attempts == 1 {
buf := make([]byte, chunkSize)
if _, err := io.ReadFull(fs, buf); err != nil {
return err
}
return errors.New("transient")
}
_, err := io.ReadAll(fs)
return err
})
snap, _, err := m.Init(initReq(user, totalSize, chunkSize))
if err != nil {
t.Fatalf("Init: %v", err)
}
sendChunk(t, m, user, snap.ID, data, 0, chunkSize)
waitState(t, m, user, snap.ID, StateFailedRetriable)
// a failed_retriable session keeps no chunk data, so identity proof is not
// required to resume: the retry-button flow works without rapid hashing,
// and the re-fill CRC check still catches a changed file
again, resumed, err := m.Init(initReq(user, totalSize, chunkSize))
if err != nil || !resumed || again.ID != snap.ID {
t.Fatalf("re-Init on failed_retriable = (id=%s, resumed=%v, err=%v), want resumed same session", again.ID, resumed, err)
}
sendChunk(t, m, user, snap.ID, data, 0, chunkSize)
sendChunk(t, m, user, snap.ID, data, 1, chunkSize)
if _, err := m.Complete(context.Background(), user, snap.ID); err != nil {
t.Fatalf("Complete after hashless refill: %v", err)
}
}
func TestRapidUploadShortCircuit(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(6 * chunkSize)
data := genData(totalSize)
user := testUser()
wantMD5 := "0123456789abcdef0123456789abcdef"
wantSHA1 := "da39a3ee5e6b4b0d3255bfef95601890afd80709"
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
// the hash provided at init must reach the driver through the stream,
// that is what lets PutRapid-style drivers skip the transfer entirely
if got := fs.GetHash().GetHash(utils.MD5); got != wantMD5 {
return fmt.Errorf("md5 not propagated: %q", got)
}
if got := fs.GetHash().GetHash(utils.SHA1); got != wantSHA1 {
return fmt.Errorf("sha1 not propagated: %q", got)
}
time.Sleep(50 * time.Millisecond) // simulated rapid-upload API round trip
up(100)
return nil // rapid upload hit: succeed without reading the stream
})
req := initReq(user, totalSize, chunkSize)
req.Hashes = map[*utils.HashType]string{utils.MD5: wantMD5, utils.SHA1: wantSHA1}
snap, _, err := m.Init(req)
if err != nil {
t.Fatalf("Init: %v", err)
}
// spam chunks across the completion moment: some land while receiving,
// some race the window close, some arrive after completion — with the
// completed-session absorption none of them may surface an error
var wg sync.WaitGroup
errCh := make(chan error, 64)
for w := 0; w < 4; w++ {
wg.Add(1)
go func(w int) {
defer wg.Done()
for round := 0; round < 8; round++ {
idx := (w*8 + round) % 6
_, err := m.Chunk(user, snap.ID, idx, bytes.NewReader(chunkOf(data, idx, chunkSize)))
// flow-control signals are part of the protocol, not failures
if err != nil && !errors.Is(err, ErrChunkInFlight) && !errors.Is(err, ErrOutOfWindow) {
errCh <- fmt.Errorf("worker %d chunk %d: %w", w, idx, err)
return
}
time.Sleep(5 * time.Millisecond)
}
}(w)
}
wg.Wait()
close(errCh)
for err := range errCh {
t.Error(err)
}
final, err := m.Complete(context.Background(), user, snap.ID)
if err != nil {
t.Fatalf("Complete: %v", err)
}
if final.State != StateCompleted || final.StorageProgress != 100 {
t.Fatalf("final snapshot = %+v", final)
}
}
// TestChunkRacesRapidCompletion pins the exact race the completed-session
// absorption exists for: a chunk request grabs the live window, stalls while
// receiving its body, the pipeline completes off the hash alone (rapid
// upload) and closes the window — the stalled chunk must then succeed
// idempotently instead of surfacing "window closed" to a client whose upload
// in fact just finished.
func TestChunkRacesRapidCompletion(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(2 * chunkSize)
data := genData(totalSize)
user := testUser()
proceed := make(chan struct{})
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
defer fs.Close()
<-proceed // the rapid-upload verdict arrives when the test says so
return nil
})
snap, _, err := m.Init(initReq(user, totalSize, chunkSize))
if err != nil {
t.Fatalf("Init: %v", err)
}
gate := make(chan struct{})
type result struct {
snap SessionSnapshot
err error
}
resCh := make(chan result, 1)
go func() {
s, e := m.Chunk(user, snap.ID, 0,
&gatedReader{release: gate, inner: bytes.NewReader(chunkOf(data, 0, chunkSize))})
resCh <- result{s, e}
}()
// wait until the chunk writer holds slot 0 in the filling state
m.mu.Lock()
sess := m.byID[snap.ID]
m.mu.Unlock()
sess.mu.Lock()
win := sess.win
sess.mu.Unlock()
deadline := time.Now().Add(5 * time.Second)
for {
win.mu.Lock()
filling := win.slotState[0] == slotFilling
win.mu.Unlock()
if filling {
break
}
if time.Now().After(deadline) {
t.Fatal("chunk writer never reached the filling state")
}
time.Sleep(time.Millisecond)
}
close(proceed) // rapid upload succeeds, window closes
waitState(t, m, user, snap.ID, StateCompleted) // completion recorded
close(gate) // stalled chunk finishes against the closed window
res := <-resCh
if res.err != nil {
t.Fatalf("in-flight chunk across rapid completion must be absorbed, got: %v", res.err)
}
if res.snap.State != StateCompleted {
t.Fatalf("absorbed chunk snapshot state = %s, want completed", res.snap.State)
}
}
// TestChunkDuringCompletionGap covers the moment between op.Put closing the
// window (its defer) and the verdict being recorded: a chunk hitting the
// closed window inside that gap must wait for the verdict and be absorbed,
// not bounce a "window closed" error at a client whose upload just succeeded.
func TestChunkDuringCompletionGap(t *testing.T) {
m := setupSessionTest(t)
const chunkSize = 1024
totalSize := int64(2 * chunkSize)
data := genData(totalSize)
user := testUser()
windowClosed := make(chan struct{})
allowVerdict := make(chan struct{})
stubPut(t, func(ctx context.Context, dst string, fs *stream.FileStream, up driver.UpdateProgress) error {
_ = fs.Close() // what op.Put's defer does before Put returns
close(windowClosed)
<-allowVerdict // hold the pipeline return open: this IS the gap
return nil
})
snap, _, err := m.Init(initReq(user, totalSize, chunkSize))
if err != nil {
t.Fatalf("Init: %v", err)
}
<-windowClosed
type result struct {
snap SessionSnapshot
err error
}
resCh := make(chan result, 1)
go func() {
s, e := m.Chunk(user, snap.ID, 0, bytes.NewReader(chunkOf(data, 0, chunkSize)))
resCh <- result{s, e}
}()
select {
case r := <-resCh:
t.Fatalf("chunk inside the gap returned early with (%s, %v); it must wait for the verdict", r.snap.State, r.err)
case <-time.After(150 * time.Millisecond):
// still waiting on the verdict, as designed
}
close(allowVerdict)
select {
case r := <-resCh:
if r.err != nil {
t.Fatalf("gap chunk must be absorbed after completion, got: %v", r.err)
}
if r.snap.State != StateCompleted {
t.Fatalf("gap chunk snapshot state = %s, want completed", r.snap.State)
}
case <-time.After(5 * time.Second):
t.Fatal("gap chunk never returned after the verdict")
}
}
func TestInitRejectsBadSize(t *testing.T) {
m := setupSessionTest(t)
if _, _, err := m.Init(initReq(testUser(), 0, 1024)); err == nil {
t.Fatal("Init with size 0: expected error")
}
}
+377
View File
@@ -0,0 +1,377 @@
package multipart
import (
"errors"
"fmt"
"hash/crc32"
"io"
"os"
"path/filepath"
"sync"
"time"
"github.com/OpenListTeam/OpenList/v4/pkg/utils"
)
const (
slotFree uint8 = iota
slotFilling
slotReady
)
var (
// ErrClosed is the sticky error after Close; all pending and future reads/writes fail with it.
ErrClosed = errors.New("multipart upload window closed")
// ErrChunkInFlight means another request is uploading the same chunk right now.
ErrChunkInFlight = errors.New("chunk is being uploaded by another request")
// ErrOutOfWindow means the chunk is still too far ahead of the consumption
// frontier after waiting WindowWaitTimeout; the client should back off and
// resend it later (flow control, not a failure).
ErrOutOfWindow = errors.New("chunk is out of the receiving window")
)
// WindowWaitTimeout bounds how long WriteChunk blocks waiting for its slot.
// Browsers cannot reliably read responses sent before the request body is
// consumed (they report a network error), so under backpressure it is far
// better to hold the request until a slot frees — the wait must just stay
// well below CDN request deadlines (Cloudflare: ~100s). Tests shrink this.
var WindowWaitTimeout = 10 * time.Second
// Window reassembles concurrently uploaded chunks into a sequential stream.
// Chunks land in a ring file of slots*chunkSize bytes (chunk i -> slot i%slots),
// and Read serves bytes in order, blocking until the next needed chunk arrives.
// A chunk slot is released as soon as the reader crosses its boundary, so the
// disk footprint is bounded by slots*chunkSize regardless of the file size.
//
// WriteChunk is safe for concurrent use; Read must be called from a single
// goroutine (the same contract as the FileStreamer it backs).
type Window struct {
mu sync.Mutex
cond *sync.Cond
f *os.File
path string
chunkSize int64
totalSize int64
total int
slots int
slotState []uint8
slotChunk []int
readPos int64
crcs []uint32
crcSet []bool
err error
}
// Snapshot describes the receiving state, used for status responses and resume.
type Snapshot struct {
// Frontier is the next chunk index to be consumed (== TotalChunks when the stream is fully consumed).
Frontier int
// ReadPos is the number of bytes already consumed by the pipeline.
ReadPos int64
// ReceivedBytes is the number of payload bytes received from the client (consumed + buffered).
ReceivedBytes int64
// Received holds inclusive ranges of chunk indexes the client does not need to resend.
Received [][2]int
}
func NewWindow(dir, id string, chunkSize, totalSize int64, slots int) (*Window, error) {
if chunkSize <= 0 || totalSize <= 0 || slots <= 0 {
return nil, fmt.Errorf("invalid window params: chunkSize=%d totalSize=%d slots=%d", chunkSize, totalSize, slots)
}
if err := os.MkdirAll(dir, 0o700); err != nil {
return nil, err
}
path := filepath.Join(dir, id+".ring")
f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0o600)
if err != nil {
return nil, err
}
total := int((totalSize + chunkSize - 1) / chunkSize)
w := &Window{
f: f,
path: path,
chunkSize: chunkSize,
totalSize: totalSize,
total: total,
slots: slots,
slotState: make([]uint8, slots),
slotChunk: make([]int, slots),
crcs: make([]uint32, total),
crcSet: make([]bool, total),
}
for i := range w.slotChunk {
w.slotChunk[i] = -1
}
w.cond = sync.NewCond(&w.mu)
return w, nil
}
func (w *Window) TotalChunks() int { return w.total }
// ChunkLen returns the payload length of chunk idx (the last chunk may be short).
func (w *Window) ChunkLen(idx int) int64 {
if idx == w.total-1 {
return w.totalSize - int64(idx)*w.chunkSize
}
return w.chunkSize
}
func stopTimer(t *time.Timer) {
if t != nil {
t.Stop()
}
}
// frontier returns the next chunk index to be consumed. Callers must hold mu.
func (w *Window) frontier() int {
if w.readPos >= w.totalSize {
return w.total
}
return int(w.readPos / w.chunkSize)
}
// WriteChunk reads exactly the chunk payload from r into the ring and returns its CRC32 (IEEE).
// Re-sending an already buffered or consumed chunk succeeds immediately without touching data.
func (w *Window) WriteChunk(idx int, r io.Reader) (uint32, error) {
w.mu.Lock()
if w.err != nil {
w.mu.Unlock()
return 0, w.err
}
if idx < 0 || idx >= w.total {
w.mu.Unlock()
return 0, fmt.Errorf("chunk index %d out of range [0,%d)", idx, w.total)
}
length := w.ChunkLen(idx)
slot := idx % w.slots
// Admission control with a bounded wait: instead of bouncing a chunk the
// moment its slot is busy, park the request until the reader frees the
// slot. Rejecting fast would answer before the request body is read, which
// browsers surface as a network error — so under backpressure, waiting IS
// the flow control. sync.Cond has no timed wait; a timer broadcast wakes
// the loop at the deadline.
deadline := time.Now().Add(WindowWaitTimeout)
var timer *time.Timer
for {
if w.err != nil {
w.mu.Unlock()
stopTimer(timer)
return 0, w.err
}
if int64(idx)*w.chunkSize+length <= w.readPos {
// already fully consumed
crc := w.crcs[idx]
w.mu.Unlock()
stopTimer(timer)
return crc, nil
}
if w.slotChunk[slot] == idx {
if w.slotState[slot] == slotReady {
crc := w.crcs[idx]
w.mu.Unlock()
stopTimer(timer)
return crc, nil
}
if w.slotState[slot] == slotFilling {
w.mu.Unlock()
stopTimer(timer)
return 0, ErrChunkInFlight
}
}
if w.slotState[slot] == slotFree && idx < w.frontier()+w.slots {
break // admissible
}
if !time.Now().Before(deadline) {
w.mu.Unlock()
stopTimer(timer)
return 0, ErrOutOfWindow
}
if timer == nil {
timer = time.AfterFunc(time.Until(deadline), func() {
w.mu.Lock()
w.cond.Broadcast()
w.mu.Unlock()
})
}
w.cond.Wait()
}
stopTimer(timer)
w.slotState[slot] = slotFilling
w.slotChunk[slot] = idx
f := w.f
w.mu.Unlock()
h := crc32.NewIEEE()
n, err := utils.CopyWithBufferN(io.NewOffsetWriter(f, int64(slot)*w.chunkSize), io.TeeReader(r, h), length)
if err == nil {
// the body must contain exactly one chunk
var b [1]byte
if m, _ := io.ReadFull(r, b[:]); m > 0 {
err = fmt.Errorf("chunk %d larger than expected %d bytes", idx, length)
}
} else {
err = fmt.Errorf("incomplete chunk %d: got %d of %d bytes: %w", idx, n, length, err)
}
w.mu.Lock()
defer w.mu.Unlock()
if w.slotState[slot] != slotFilling || w.slotChunk[slot] != idx {
// the window was closed and reset the slot while we were writing
if w.err != nil {
return 0, w.err
}
return 0, ErrClosed
}
if err == nil && w.err != nil {
err = w.err
}
if err != nil {
w.slotState[slot] = slotFree
w.slotChunk[slot] = -1
return 0, err
}
w.slotState[slot] = slotReady
w.crcs[idx] = h.Sum32()
w.crcSet[idx] = true
w.cond.Broadcast()
return w.crcs[idx], nil
}
// Read serves the reassembled stream in order, blocking until the next chunk
// is available, the window is closed, or the stream ends (io.EOF).
func (w *Window) Read(p []byte) (int, error) {
if len(p) == 0 {
return 0, nil
}
w.mu.Lock()
for {
if w.err != nil {
w.mu.Unlock()
return 0, w.err
}
if w.readPos >= w.totalSize {
w.mu.Unlock()
return 0, io.EOF
}
cur := int(w.readPos / w.chunkSize)
slot := cur % w.slots
if w.slotState[slot] == slotReady && w.slotChunk[slot] == cur {
chunkStart := int64(cur) * w.chunkSize
chunkEnd := chunkStart + w.ChunkLen(cur)
n := int64(len(p))
if avail := chunkEnd - w.readPos; n > avail {
n = avail
}
off := int64(slot)*w.chunkSize + (w.readPos - chunkStart)
f := w.f
w.mu.Unlock()
read, err := f.ReadAt(p[:n], off)
w.mu.Lock()
if read > 0 {
w.readPos += int64(read)
if w.readPos >= chunkEnd {
w.slotState[slot] = slotFree
w.slotChunk[slot] = -1
w.cond.Broadcast() // writers may be parked waiting for this slot
}
}
sticky := w.err
w.mu.Unlock()
if read > 0 {
return read, nil
}
if sticky != nil {
return 0, sticky
}
if err == nil {
err = io.ErrUnexpectedEOF
}
return 0, err
}
w.cond.Wait()
}
}
// Close makes all pending and future operations fail with ErrClosed and removes
// the ring file. It is invoked by op.Put via FileStream.Closers when the
// pipeline ends, and is safe to call multiple times.
func (w *Window) Close() error {
return w.CloseWithError(ErrClosed)
}
// CloseWithError is Close with a caller-chosen sticky error. The session's
// abort path passes an error wrapping context.Canceled so that a driver woken
// up from a blocked Read treats the abort exactly like a canceled request
// (e.g. the local driver only removes partially written files in that case).
func (w *Window) CloseWithError(sticky error) error {
w.mu.Lock()
if w.err == nil {
w.err = sticky
}
f := w.f
w.f = nil
for i := range w.slotState {
w.slotState[i] = slotFree
w.slotChunk[i] = -1
}
w.cond.Broadcast()
w.mu.Unlock()
if f == nil {
return nil
}
err := f.Close()
if rmErr := os.Remove(w.path); rmErr != nil && err == nil {
err = rmErr
}
return err
}
// Snapshot reports the receiving state for status responses and resume discovery.
func (w *Window) Snapshot() Snapshot {
w.mu.Lock()
defer w.mu.Unlock()
snap := Snapshot{Frontier: w.frontier(), ReadPos: w.readPos, ReceivedBytes: w.readPos}
var ranges [][2]int
if snap.Frontier > 0 {
ranges = append(ranges, [2]int{0, snap.Frontier - 1})
}
for idx := snap.Frontier; idx < snap.Frontier+w.slots && idx < w.total; idx++ {
slot := idx % w.slots
if w.slotState[slot] != slotReady || w.slotChunk[slot] != idx {
continue
}
start := int64(idx) * w.chunkSize
var consumed int64
if w.readPos > start {
consumed = w.readPos - start
}
snap.ReceivedBytes += w.ChunkLen(idx) - consumed
if len(ranges) > 0 && ranges[len(ranges)-1][1] == idx-1 {
ranges[len(ranges)-1][1] = idx
} else {
ranges = append(ranges, [2]int{idx, idx})
}
}
snap.Received = ranges
return snap
}
// CRCs returns a copy of the per-chunk CRC32 table and which entries are set.
// It remains readable after Close, so the session can compare re-filled chunks
// against a previous attempt.
func (w *Window) CRCs() ([]uint32, []bool) {
w.mu.Lock()
defer w.mu.Unlock()
crcs := make([]uint32, len(w.crcs))
set := make([]bool, len(w.crcSet))
copy(crcs, w.crcs)
copy(set, w.crcSet)
return crcs, set
}
+527
View File
@@ -0,0 +1,527 @@
package multipart
import (
"bytes"
"crypto/sha256"
"errors"
"hash/crc32"
"io"
"math/rand"
"sync"
"sync/atomic"
"testing"
"time"
)
func genData(size int64) []byte {
data := make([]byte, size)
rnd := rand.New(rand.NewSource(size*7919 + 13))
rnd.Read(data)
return data
}
func newTestWindow(t *testing.T, chunkSize, totalSize int64, slots int) *Window {
t.Helper()
w, err := NewWindow(t.TempDir(), "test", chunkSize, totalSize, slots)
if err != nil {
t.Fatalf("NewWindow: %v", err)
}
t.Cleanup(func() { _ = w.Close() })
return w
}
func chunkOf(data []byte, idx int, chunkSize int64) []byte {
start := int64(idx) * chunkSize
end := start + chunkSize
if end > int64(len(data)) {
end = int64(len(data))
}
return data[start:end]
}
func writeChunkOK(t *testing.T, w *Window, data []byte, idx int) uint32 {
t.Helper()
crc, err := w.WriteChunk(idx, bytes.NewReader(chunkOf(data, idx, w.chunkSize)))
if err != nil {
t.Fatalf("WriteChunk(%d): %v", idx, err)
}
if want := crc32.ChecksumIEEE(chunkOf(data, idx, w.chunkSize)); crc != want {
t.Fatalf("WriteChunk(%d) crc = %08x, want %08x", idx, crc, want)
}
return crc
}
// readAllWithin reads the whole stream in a goroutine and fails the test on timeout,
// so a reassembly bug cannot hang the suite.
func readAllWithin(t *testing.T, w *Window, timeout time.Duration) []byte {
t.Helper()
type result struct {
data []byte
err error
}
ch := make(chan result, 1)
go func() {
data, err := io.ReadAll(w)
ch <- result{data, err}
}()
select {
case res := <-ch:
if res.err != nil {
t.Fatalf("ReadAll: %v", res.err)
}
return res.data
case <-time.After(timeout):
t.Fatal("ReadAll timed out")
return nil
}
}
func TestSequentialReadWrite(t *testing.T) {
const chunkSize = 64 * 1024
totalSize := int64(4*chunkSize + 32*1024) // last chunk is short
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 8)
go func() {
for i := 0; i < w.TotalChunks(); i++ {
if _, err := w.WriteChunk(i, bytes.NewReader(chunkOf(data, i, chunkSize))); err != nil {
t.Errorf("WriteChunk(%d): %v", i, err)
return
}
time.Sleep(time.Millisecond)
}
}()
got := readAllWithin(t, w, 10*time.Second)
if !bytes.Equal(got, data) {
t.Fatalf("reassembled stream differs: got %d bytes, want %d", len(got), len(data))
}
}
func TestOutOfOrderWrites(t *testing.T) {
const chunkSize = 16 * 1024
totalSize := int64(5*chunkSize - 100)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 8)
for _, idx := range []int{3, 0, 4, 2, 1} {
writeChunkOK(t, w, data, idx)
}
got := readAllWithin(t, w, 10*time.Second)
if !bytes.Equal(got, data) {
t.Fatal("reassembled stream differs after out-of-order writes")
}
}
func TestConcurrentWriters(t *testing.T) {
const chunkSize = 32 * 1024
totalSize := int64(8 * chunkSize)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 8)
var wg sync.WaitGroup
for i := 0; i < w.TotalChunks(); i++ {
wg.Add(1)
go func(idx int) {
defer wg.Done()
if _, err := w.WriteChunk(idx, bytes.NewReader(chunkOf(data, idx, chunkSize))); err != nil {
t.Errorf("WriteChunk(%d): %v", idx, err)
}
}(i)
}
got := readAllWithin(t, w, 10*time.Second)
wg.Wait()
if !bytes.Equal(got, data) {
t.Fatal("reassembled stream differs after concurrent writes")
}
}
func setWaitTimeout(t *testing.T, d time.Duration) {
t.Helper()
old := WindowWaitTimeout
WindowWaitTimeout = d
t.Cleanup(func() { WindowWaitTimeout = old })
}
func TestBackpressure(t *testing.T) {
setWaitTimeout(t, 50*time.Millisecond) // assert the post-deadline rejection
const chunkSize = 1024
totalSize := int64(5 * chunkSize)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 2)
if _, err := w.WriteChunk(2, bytes.NewReader(chunkOf(data, 2, chunkSize))); !errors.Is(err, ErrOutOfWindow) {
t.Fatalf("chunk 2 with frontier 0: err = %v, want ErrOutOfWindow", err)
}
writeChunkOK(t, w, data, 0)
writeChunkOK(t, w, data, 1)
// both slots occupied, frontier still 0
if _, err := w.WriteChunk(2, bytes.NewReader(chunkOf(data, 2, chunkSize))); !errors.Is(err, ErrOutOfWindow) {
t.Fatalf("chunk 2 with full window: err = %v, want ErrOutOfWindow", err)
}
// consume chunk 0 -> slot released, frontier advances
buf := make([]byte, chunkSize)
if _, err := io.ReadFull(w, buf); err != nil {
t.Fatalf("ReadFull chunk 0: %v", err)
}
if !bytes.Equal(buf, chunkOf(data, 0, chunkSize)) {
t.Fatal("chunk 0 content differs")
}
writeChunkOK(t, w, data, 2)
// chunk 3 maps to the slot still holding buffered chunk 1
if _, err := w.WriteChunk(3, bytes.NewReader(chunkOf(data, 3, chunkSize))); !errors.Is(err, ErrOutOfWindow) {
t.Fatalf("chunk 3 with occupied slot: err = %v, want ErrOutOfWindow", err)
}
}
// TestWriteChunkWaitsForSlot pins the browser-friendly flow control: a chunk
// whose slot is occupied parks until the reader frees it instead of bouncing
// with an immediate rejection (early responses read as network errors in
// browsers).
func TestWriteChunkWaitsForSlot(t *testing.T) {
setWaitTimeout(t, 5*time.Second)
const chunkSize = 1024
totalSize := int64(5 * chunkSize)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 2)
writeChunkOK(t, w, data, 0)
writeChunkOK(t, w, data, 1)
done := make(chan error, 1)
go func() {
_, err := w.WriteChunk(2, bytes.NewReader(chunkOf(data, 2, chunkSize)))
done <- err
}()
select {
case err := <-done:
t.Fatalf("chunk 2 should be parked while the window is full, returned %v", err)
case <-time.After(100 * time.Millisecond):
// parked, as designed
}
// consuming chunk 0 frees its slot and must wake the parked writer
buf := make([]byte, chunkSize)
if _, err := io.ReadFull(w, buf); err != nil {
t.Fatalf("ReadFull: %v", err)
}
select {
case err := <-done:
if err != nil {
t.Fatalf("parked chunk after slot freed: %v", err)
}
case <-time.After(5 * time.Second):
t.Fatal("parked chunk never admitted after its slot freed")
}
}
func TestIdempotentResend(t *testing.T) {
const chunkSize = 1024
totalSize := int64(2 * chunkSize)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 4)
first := writeChunkOK(t, w, data, 0)
again := writeChunkOK(t, w, data, 0) // buffered, not yet consumed
if first != again {
t.Fatalf("resend crc = %08x, want %08x", again, first)
}
buf := make([]byte, chunkSize)
if _, err := io.ReadFull(w, buf); err != nil {
t.Fatalf("ReadFull: %v", err)
}
consumed := writeChunkOK(t, w, data, 0) // already consumed
if consumed != first {
t.Fatalf("post-consume resend crc = %08x, want %08x", consumed, first)
}
}
// gatedReader blocks the first Read until released, to hold a chunk in the filling state.
type gatedReader struct {
release <-chan struct{}
inner io.Reader
}
func (g *gatedReader) Read(p []byte) (int, error) {
<-g.release
return g.inner.Read(p)
}
func TestInFlightConflict(t *testing.T) {
const chunkSize = 1024
totalSize := int64(2 * chunkSize)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 4)
release := make(chan struct{})
done := make(chan error, 1)
go func() {
_, err := w.WriteChunk(0, &gatedReader{release: release, inner: bytes.NewReader(chunkOf(data, 0, chunkSize))})
done <- err
}()
// wait until the writer marked the slot as filling
deadline := time.Now().Add(5 * time.Second)
for {
w.mu.Lock()
filling := w.slotState[0] == slotFilling
w.mu.Unlock()
if filling {
break
}
if time.Now().After(deadline) {
t.Fatal("writer never reached filling state")
}
time.Sleep(time.Millisecond)
}
if _, err := w.WriteChunk(0, bytes.NewReader(chunkOf(data, 0, chunkSize))); !errors.Is(err, ErrChunkInFlight) {
t.Fatalf("concurrent same-chunk write: err = %v, want ErrChunkInFlight", err)
}
close(release)
if err := <-done; err != nil {
t.Fatalf("gated WriteChunk: %v", err)
}
writeChunkOK(t, w, data, 0) // idempotent after settle
}
func TestShortBodyRecovers(t *testing.T) {
const chunkSize = 1024
totalSize := int64(2 * chunkSize)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 4)
if _, err := w.WriteChunk(0, bytes.NewReader(chunkOf(data, 0, chunkSize)[:100])); err == nil {
t.Fatal("short body: expected error")
}
writeChunkOK(t, w, data, 0) // slot must have been recycled
}
func TestOversizeBodyRejected(t *testing.T) {
const chunkSize = 1024
totalSize := int64(2 * chunkSize)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 4)
oversize := append(append([]byte{}, chunkOf(data, 0, chunkSize)...), 0xFF)
if _, err := w.WriteChunk(0, bytes.NewReader(oversize)); err == nil {
t.Fatal("oversize body: expected error")
}
writeChunkOK(t, w, data, 0)
// the short last chunk must also reject a full-size body
last := w.TotalChunks() - 1
if last == 0 {
t.Fatal("test needs at least 2 chunks")
}
}
func TestLastChunkShortStrict(t *testing.T) {
const chunkSize = 1024
totalSize := int64(chunkSize + 100)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 4)
if _, err := w.WriteChunk(1, bytes.NewReader(genData(chunkSize))); err == nil {
t.Fatal("full-size body for short last chunk: expected error")
}
writeChunkOK(t, w, data, 1)
writeChunkOK(t, w, data, 0)
got := readAllWithin(t, w, 10*time.Second)
if !bytes.Equal(got, data) {
t.Fatal("reassembled stream differs")
}
}
func TestCloseUnblocksReader(t *testing.T) {
w := newTestWindow(t, 1024, 4096, 4)
errCh := make(chan error, 1)
go func() {
buf := make([]byte, 16)
_, err := w.Read(buf)
errCh <- err
}()
time.Sleep(20 * time.Millisecond)
_ = w.Close()
select {
case err := <-errCh:
if !errors.Is(err, ErrClosed) {
t.Fatalf("blocked Read after Close: err = %v, want ErrClosed", err)
}
case <-time.After(5 * time.Second):
t.Fatal("Read still blocked after Close")
}
}
func TestCloseWithErrorPropagatesSticky(t *testing.T) {
w := newTestWindow(t, 1024, 4096, 4)
cause := errors.New("aborted for a reason")
errCh := make(chan error, 1)
go func() {
buf := make([]byte, 16)
_, err := w.Read(buf)
errCh <- err
}()
time.Sleep(20 * time.Millisecond)
_ = w.CloseWithError(cause)
select {
case err := <-errCh:
if !errors.Is(err, cause) {
t.Fatalf("blocked Read after CloseWithError: err = %v, want %v", err, cause)
}
case <-time.After(5 * time.Second):
t.Fatal("Read still blocked after CloseWithError")
}
if _, err := w.WriteChunk(0, bytes.NewReader(make([]byte, 1024))); !errors.Is(err, cause) {
t.Fatalf("WriteChunk after CloseWithError: err = %v, want %v", err, cause)
}
}
func TestWriteAfterClose(t *testing.T) {
w := newTestWindow(t, 1024, 4096, 4)
_ = w.Close()
if _, err := w.WriteChunk(0, bytes.NewReader(make([]byte, 1024))); !errors.Is(err, ErrClosed) {
t.Fatalf("WriteChunk after Close: err = %v, want ErrClosed", err)
}
}
func TestCloseUnblocksInFlightWriter(t *testing.T) {
const chunkSize = 1024
data := genData(2 * chunkSize)
w := newTestWindow(t, chunkSize, 2*chunkSize, 4)
release := make(chan struct{})
done := make(chan error, 1)
go func() {
_, err := w.WriteChunk(0, &gatedReader{release: release, inner: bytes.NewReader(chunkOf(data, 0, chunkSize))})
done <- err
}()
time.Sleep(20 * time.Millisecond)
_ = w.Close()
close(release)
if err := <-done; !errors.Is(err, ErrClosed) {
t.Fatalf("in-flight WriteChunk across Close: err = %v, want ErrClosed", err)
}
}
func TestEOFExact(t *testing.T) {
const chunkSize = 1024
totalSize := int64(chunkSize + 5)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 4)
writeChunkOK(t, w, data, 0)
writeChunkOK(t, w, data, 1)
got := readAllWithin(t, w, 10*time.Second)
if !bytes.Equal(got, data) {
t.Fatal("reassembled stream differs")
}
buf := make([]byte, 1)
if n, err := w.Read(buf); n != 0 || err != io.EOF {
t.Fatalf("Read at EOF = (%d, %v), want (0, io.EOF)", n, err)
}
}
func TestManyLapsSmallWindow(t *testing.T) {
const chunkSize = 8 * 1024
const chunks = 64
totalSize := int64(chunks*chunkSize - 777)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 3)
var next atomic.Int64
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for {
idx := int(next.Add(1) - 1)
if idx >= w.TotalChunks() {
return
}
for {
_, err := w.WriteChunk(idx, bytes.NewReader(chunkOf(data, idx, chunkSize)))
if err == nil {
break
}
if errors.Is(err, ErrOutOfWindow) || errors.Is(err, ErrChunkInFlight) {
time.Sleep(200 * time.Microsecond)
continue
}
t.Errorf("WriteChunk(%d): %v", idx, err)
return
}
}
}()
}
got := readAllWithin(t, w, 30*time.Second)
wg.Wait()
if wantSum, gotSum := sha256.Sum256(data), sha256.Sum256(got); wantSum != gotSum {
t.Fatalf("reassembled stream differs: got %d bytes, want %d", len(got), len(data))
}
}
func TestSnapshot(t *testing.T) {
const chunkSize = 1024
totalSize := int64(5*chunkSize + 512)
data := genData(totalSize)
w := newTestWindow(t, chunkSize, totalSize, 4)
writeChunkOK(t, w, data, 0)
writeChunkOK(t, w, data, 1)
writeChunkOK(t, w, data, 3)
snap := w.Snapshot()
if snap.Frontier != 0 || snap.ReadPos != 0 {
t.Fatalf("snapshot frontier/readPos = %d/%d, want 0/0", snap.Frontier, snap.ReadPos)
}
wantRanges := [][2]int{{0, 1}, {3, 3}}
if len(snap.Received) != len(wantRanges) || snap.Received[0] != wantRanges[0] || snap.Received[1] != wantRanges[1] {
t.Fatalf("snapshot received = %v, want %v", snap.Received, wantRanges)
}
if snap.ReceivedBytes != 3*chunkSize {
t.Fatalf("snapshot receivedBytes = %d, want %d", snap.ReceivedBytes, 3*chunkSize)
}
buf := make([]byte, chunkSize)
if _, err := io.ReadFull(w, buf); err != nil {
t.Fatalf("ReadFull: %v", err)
}
snap = w.Snapshot()
if snap.Frontier != 1 || snap.ReadPos != chunkSize {
t.Fatalf("snapshot frontier/readPos = %d/%d, want 1/%d", snap.Frontier, snap.ReadPos, chunkSize)
}
if len(snap.Received) != 2 || snap.Received[0] != [2]int{0, 1} || snap.Received[1] != [2]int{3, 3} {
t.Fatalf("snapshot received = %v, want [[0,1],[3,3]]", snap.Received)
}
if snap.ReceivedBytes != 3*chunkSize {
t.Fatalf("snapshot receivedBytes = %d, want %d", snap.ReceivedBytes, 3*chunkSize)
}
}
func TestIndexOutOfRange(t *testing.T) {
w := newTestWindow(t, 1024, 4096, 4)
if _, err := w.WriteChunk(-1, bytes.NewReader(nil)); err == nil {
t.Fatal("negative index: expected error")
}
if _, err := w.WriteChunk(4, bytes.NewReader(nil)); err == nil {
t.Fatal("index == total: expected error")
}
}
func TestCRCsSurviveClose(t *testing.T) {
const chunkSize = 1024
data := genData(2 * chunkSize)
w := newTestWindow(t, chunkSize, 2*chunkSize, 4)
want := writeChunkOK(t, w, data, 0)
_ = w.Close()
crcs, set := w.CRCs()
if !set[0] || crcs[0] != want {
t.Fatalf("CRCs after Close = (%08x, %v), want (%08x, true)", crcs[0], set[0], want)
}
if set[1] {
t.Fatal("chunk 1 crc should not be set")
}
}
+1
View File
@@ -6,6 +6,7 @@ import (
_ "github.com/OpenListTeam/OpenList/v4/internal/offline_download/123"
_ "github.com/OpenListTeam/OpenList/v4/internal/offline_download/123_open"
_ "github.com/OpenListTeam/OpenList/v4/internal/offline_download/aria2"
_ "github.com/OpenListTeam/OpenList/v4/internal/offline_download/guangyapan"
_ "github.com/OpenListTeam/OpenList/v4/internal/offline_download/http"
_ "github.com/OpenListTeam/OpenList/v4/internal/offline_download/pikpak"
_ "github.com/OpenListTeam/OpenList/v4/internal/offline_download/qbit"
@@ -0,0 +1,132 @@
package guangyapan
import (
"context"
"errors"
"fmt"
"sync/atomic"
guangyapandriver "github.com/OpenListTeam/OpenList/v4/drivers/guangyapan"
"github.com/OpenListTeam/OpenList/v4/internal/conf"
"github.com/OpenListTeam/OpenList/v4/internal/errs"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/offline_download/tool"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/internal/setting"
)
type GuangYaPan struct {
refreshTaskCache atomic.Bool
}
func (g *GuangYaPan) Name() string {
return "GuangYaPan"
}
func (g *GuangYaPan) Items() []model.SettingItem {
return nil
}
func (g *GuangYaPan) Run(task *tool.DownloadTask) error {
return errs.NotSupport
}
func (g *GuangYaPan) Init() (string, error) {
g.refreshTaskCache.Store(false)
return "ok", nil
}
func (g *GuangYaPan) IsReady() bool {
tempDir := setting.GetStr(conf.GuangYaPanTempDir)
if tempDir == "" {
return false
}
storage, _, err := op.GetStorageAndActualPath(tempDir)
if err != nil {
return false
}
if _, ok := storage.(*guangyapandriver.GuangYaPan); !ok {
return false
}
return true
}
func (g *GuangYaPan) AddURL(args *tool.AddUrlArgs) (string, error) {
g.refreshTaskCache.Store(true)
storage, actualPath, err := op.GetStorageAndActualPath(args.TempDir)
if err != nil {
return "", err
}
driver, ok := storage.(*guangyapandriver.GuangYaPan)
if !ok {
return "", errors.New("GuangYaPan offline download only supports GuangYaPan destination storage")
}
ctx := context.Background()
if err := op.MakeDir(ctx, storage, actualPath); err != nil {
return "", err
}
parentDir, err := op.GetUnwrap(ctx, storage, actualPath)
if err != nil {
return "", err
}
task, err := driver.OfflineDownload(ctx, args.Url, parentDir, "")
if err != nil {
return "", fmt.Errorf("failed to add offline download task: %w", err)
}
return task.TaskID, nil
}
func (g *GuangYaPan) Remove(task *tool.DownloadTask) error {
storage, _, err := op.GetStorageAndActualPath(task.TempDir)
if err != nil {
return err
}
driver, ok := storage.(*guangyapandriver.GuangYaPan)
if !ok {
return errors.New("GuangYaPan offline download only supports GuangYaPan destination storage")
}
ctx := context.Background()
if err := driver.DeleteOfflineTasks(ctx, []string{task.GID}); err != nil {
return err
}
g.DelTaskCache(driver, task.GID)
return nil
}
func (g *GuangYaPan) Status(task *tool.DownloadTask) (*tool.Status, error) {
storage, _, err := op.GetStorageAndActualPath(task.TempDir)
if err != nil {
return nil, err
}
driver, ok := storage.(*guangyapandriver.GuangYaPan)
if !ok {
return nil, errors.New("GuangYaPan offline download only supports GuangYaPan destination storage")
}
tasks, err := g.GetTasks(driver, task.GID)
if err != nil {
return nil, err
}
status := &tool.Status{
Status: "the task has been deleted",
}
for _, t := range tasks {
if t.TaskID != task.GID {
continue
}
status.Progress = float64(t.Progress)
status.TotalBytes = t.TotalSize
status.Completed = t.Status == offlineStatusCompleted || t.Status == offlineStatusPartiallyCompleted
status.Status = taskStatusText(t)
if t.Status == offlineStatusFailed || t.Status == offlineStatusCanceled {
status.Err = errors.New(status.Status)
}
return status, nil
}
status.Err = errors.New("the task has been deleted")
return status, nil
}
func init() {
tool.Tools.Add(&GuangYaPan{})
}
@@ -0,0 +1,77 @@
package guangyapan
import (
"context"
"fmt"
"time"
guangyapandriver "github.com/OpenListTeam/OpenList/v4/drivers/guangyapan"
"github.com/OpenListTeam/OpenList/v4/internal/op"
"github.com/OpenListTeam/OpenList/v4/pkg/singleflight"
"github.com/OpenListTeam/go-cache"
)
const (
offlineStatusQueued = 0
offlineStatusRunning = 1
offlineStatusCompleted = 2
offlineStatusFailed = 3
offlineStatusCanceled = 4
offlineStatusPartiallyCompleted = 5
)
var taskCache = cache.NewMemCache(cache.WithShards[[]guangyapandriver.OfflineTask](16))
var taskG singleflight.Group[[]guangyapandriver.OfflineTask]
func (g *GuangYaPan) GetTasks(driver *guangyapandriver.GuangYaPan, taskID string) ([]guangyapandriver.OfflineTask, error) {
key := op.Key(driver, "/cloudcollection/v1/list_task/"+taskID)
if !g.refreshTaskCache.Load() {
if tasks, ok := taskCache.Get(key); ok {
return tasks, nil
}
}
g.refreshTaskCache.Store(false)
tasks, err, _ := taskG.Do(key, func() ([]guangyapandriver.OfflineTask, error) {
ctx := context.Background()
tasks, err := driver.OfflineList(ctx, []string{taskID}, nil, "", 0)
if err != nil {
return nil, err
}
if len(tasks) > 0 {
taskCache.Set(key, tasks, cache.WithEx[[]guangyapandriver.OfflineTask](time.Second*10))
} else {
taskCache.Del(key)
}
return tasks, nil
})
if err != nil {
return nil, err
}
return tasks, nil
}
func (g *GuangYaPan) DelTaskCache(driver *guangyapandriver.GuangYaPan, taskID string) {
taskCache.Del(op.Key(driver, "/cloudcollection/v1/list_task/"+taskID))
}
func taskStatusText(task guangyapandriver.OfflineTask) string {
switch task.Status {
case offlineStatusQueued:
return "queued"
case offlineStatusRunning:
if task.Progress > 0 {
return fmt.Sprintf("running (%d%%)", task.Progress)
}
return "running"
case offlineStatusCompleted:
return "completed"
case offlineStatusFailed:
return "failed"
case offlineStatusCanceled:
return "canceled"
case offlineStatusPartiallyCompleted:
return "partially completed"
default:
return fmt.Sprintf("unknown status %d", task.Status)
}
}
+11
View File
@@ -12,6 +12,7 @@ import (
_115_open "github.com/OpenListTeam/OpenList/v4/drivers/115_open"
_123 "github.com/OpenListTeam/OpenList/v4/drivers/123"
_123_open "github.com/OpenListTeam/OpenList/v4/drivers/123_open"
"github.com/OpenListTeam/OpenList/v4/drivers/guangyapan"
"github.com/OpenListTeam/OpenList/v4/drivers/pikpak"
"github.com/OpenListTeam/OpenList/v4/drivers/thunder"
"github.com/OpenListTeam/OpenList/v4/drivers/thunder_browser"
@@ -162,6 +163,16 @@ func AddURL(ctx context.Context, args *AddURLArgs) (task.TaskExtensionInfo, erro
} else {
tempDir = filepath.Join(setting.GetStr(conf.ThunderXTempDir), uid)
}
case "GuangYaPan":
if _, ok := storage.(*guangyapan.GuangYaPan); ok {
tempDir = args.DstDirPath
} else {
tempBase := setting.GetStr(conf.GuangYaPanTempDir)
if tempBase == "" {
return nil, errors.New("GuangYaPan temp dir is not set")
}
tempDir = filepath.Join(tempBase, uid)
}
}
taskCreator, _ := ctx.Value(conf.UserKey).(*model.User) // taskCreator is nil when convert failed
+4 -1
View File
@@ -98,6 +98,9 @@ outer:
if t.tool.Name() == "ThunderX" {
return nil
}
if t.tool.Name() == "GuangYaPan" {
return nil
}
if t.tool.Name() == "115 Cloud" {
// hack for 115
<-time.After(time.Second * 1)
@@ -176,7 +179,7 @@ func (t *DownloadTask) Update() (bool, error) {
func (t *DownloadTask) Transfer() error {
toolName := t.tool.Name()
if toolName == "115 Cloud" || toolName == "115 Open" || toolName == "123 Open" || toolName == "123Pan" || toolName == "PikPak" || toolName == "Thunder" || toolName == "ThunderX" || toolName == "ThunderBrowser" {
if toolName == "115 Cloud" || toolName == "115 Open" || toolName == "123 Open" || toolName == "123Pan" || toolName == "PikPak" || toolName == "Thunder" || toolName == "ThunderX" || toolName == "ThunderBrowser" || toolName == "GuangYaPan" {
// 如果不是直接下载到目标路径,则进行转存
if t.TempDir != t.DstDirPath {
return transferObj(t.Ctx(), t.TempDir, t.DstDirPath, t.DeletePolicy)
+13
View File
@@ -2,7 +2,9 @@ package op
import (
"context"
"fmt"
"regexp"
"strconv"
"strings"
"github.com/OpenListTeam/OpenList/v4/internal/conf"
@@ -83,6 +85,17 @@ var settingItemHooks = map[string]SettingItemHook{
conf.SlicesMap[conf.IgnoreDirectLinkParams] = strings.Split(item.Value, ",")
return nil
},
conf.MultipartChunkSize: func(item *model.SettingItem) error {
size, err := strconv.Atoi(strings.TrimSpace(item.Value))
if err != nil || size < 1 {
// deliberately a plain error: SaveSettings formats hook errors
// with %+v, which would dump a full stack trace into the UI
// notification for stack-carrying errors
return fmt.Errorf("multipart chunk size must be a positive integer (MB), got %q", item.Value)
}
item.Value = strconv.Itoa(size)
return nil
},
}
func RegisterSettingItemHook(key string, hook SettingItemHook) {
+10
View File
@@ -91,6 +91,16 @@ func GetSharingUnwrapPath(sharing *model.Sharing, path string) (unwrapPath strin
if len(sharing.Files) == 0 {
return "", errors.New("cannot get actual path of an invalid sharing")
}
// Re-validate that the shared paths are still within the creator's current
// BasePath. This prevents access to files that fell out-of-scope after the
// creator's BasePath was changed by an admin.
if sharing.Creator != nil && !sharing.Creator.IsAdmin() {
for _, f := range sharing.Files {
if !utils.IsSubPath(sharing.Creator.BasePath, f) {
return "", errors.Errorf("sharing path [%s] is outside the creator's base path", f)
}
}
}
if len(sharing.Files) == 1 {
return stdpath.Join(sharing.Files[0], path), nil
}
+70 -18
View File
@@ -26,18 +26,9 @@ func (b *Bleve) Config() searcher.Config {
}
func (b *Bleve) Search(ctx context.Context, req model.SearchReq) ([]model.SearchNode, int64, error) {
var queries []query2.Query
query := bleve.NewMatchQuery(req.Keywords)
query.SetField("name")
queries = append(queries, query)
if req.Scope != 0 {
isDir := req.Scope == 1
isDirQuery := bleve.NewBoolFieldQuery(isDir)
queries = append(queries, isDirQuery)
}
reqQuery := bleve.NewConjunctionQuery(queries...)
reqQuery := buildQuery(req)
search := bleve.NewSearchRequest(reqQuery)
search.SortBy([]string{"name"})
search.SortBy([]string{"name", "_id"})
search.From = (req.Page - 1) * req.PerPage
search.Size = req.PerPage
search.Fields = []string{"*"}
@@ -47,14 +38,74 @@ func (b *Bleve) Search(ctx context.Context, req model.SearchReq) ([]model.Search
return nil, 0, err
}
res, err := utils.SliceConvert(searchResults.Hits, func(src *search2.DocumentMatch) (model.SearchNode, error) {
return model.SearchNode{
Parent: src.Fields["parent"].(string),
Name: src.Fields["name"].(string),
IsDir: src.Fields["is_dir"].(bool),
Size: int64(src.Fields["size"].(float64)),
}, nil
return searchNodeFromHit(src), nil
})
return res, int64(searchResults.Total), nil
return res, int64(searchResults.Total), err
}
const searchBatchSize = 1000
func (b *Bleve) SearchFiltered(ctx context.Context, req model.SearchReq, filter searcher.Filter) ([]model.SearchNode, int64, error) {
reqQuery := buildQuery(req)
from := int64(req.Page-1) * int64(req.PerPage)
to := from + int64(req.PerPage)
var (
result []model.SearchNode
total int64
searchAfter []string
)
for {
search := bleve.NewSearchRequest(reqQuery)
search.SortBy([]string{"name", "_id"})
search.Size = searchBatchSize
search.Fields = []string{"*"}
if searchAfter != nil {
search.SetSearchAfter(searchAfter)
}
searchResults, err := b.BIndex.Search(search)
if err != nil {
log.Errorf("search error: %+v", err)
return nil, 0, err
}
for _, hit := range searchResults.Hits {
node := searchNodeFromHit(hit)
if !utils.IsSubPath(req.Parent, node.Parent) || filter != nil && !filter(node) {
continue
}
if total >= from && total < to {
result = append(result, node)
}
total++
}
if len(searchResults.Hits) < searchBatchSize {
break
}
last := searchResults.Hits[len(searchResults.Hits)-1]
searchAfter = append(searchAfter[:0], last.Sort...)
}
return result, total, nil
}
func buildQuery(req model.SearchReq) query2.Query {
var queries []query2.Query
query := bleve.NewMatchQuery(req.Keywords)
query.SetField("name")
queries = append(queries, query)
if req.Scope != 0 {
isDir := req.Scope == 1
isDirQuery := bleve.NewBoolFieldQuery(isDir)
queries = append(queries, isDirQuery)
}
return bleve.NewConjunctionQuery(queries...)
}
func searchNodeFromHit(src *search2.DocumentMatch) model.SearchNode {
return model.SearchNode{
Parent: src.Fields["parent"].(string),
Name: src.Fields["name"].(string),
IsDir: src.Fields["is_dir"].(bool),
Size: int64(src.Fields["size"].(float64)),
}
}
func (b *Bleve) Index(ctx context.Context, node model.SearchNode) error {
@@ -103,3 +154,4 @@ func (b *Bleve) Clear(ctx context.Context) error {
}
var _ searcher.Searcher = (*Bleve)(nil)
var _ searcher.FilteredSearcher = (*Bleve)(nil)
+52
View File
@@ -0,0 +1,52 @@
package bleve
import (
"context"
"fmt"
"testing"
"github.com/OpenListTeam/OpenList/v4/internal/model"
blevelib "github.com/blevesearch/bleve/v2"
)
func TestSearchFilteredKeepsDuplicateSortValuesAcrossBatches(t *testing.T) {
indexMapping := blevelib.NewIndexMapping()
searchNodeMapping := blevelib.NewDocumentMapping()
searchNodeMapping.AddFieldMappingsAt("is_dir", blevelib.NewBooleanFieldMapping())
searchNodeMapping.AddFieldMappingsAt("parent", blevelib.NewTextFieldMapping())
searchNodeMapping.AddFieldMappingsAt("name", blevelib.NewKeywordFieldMapping())
indexMapping.AddDocumentMapping("SearchNode", searchNodeMapping)
index, err := blevelib.NewMemOnly(indexMapping)
if err != nil {
t.Fatalf("NewMemOnly() error = %v", err)
}
t.Cleanup(func() { _ = index.Close() })
batch := index.NewBatch()
for i := 0; i < searchBatchSize+1; i++ {
batch.Index(fmt.Sprintf("allowed-%04d", i), model.SearchNode{
Parent: "/base",
Name: "duplicate",
})
}
batch.Index("denied", model.SearchNode{Parent: "/base2", Name: "duplicate"})
if err := index.Batch(batch); err != nil {
t.Fatalf("Batch() error = %v", err)
}
b := &Bleve{BIndex: index}
nodes, total, err := b.SearchFiltered(context.Background(), model.SearchReq{
Parent: "/base",
Keywords: "duplicate",
PageReq: model.PageReq{Page: 1, PerPage: searchBatchSize + 1},
}, nil)
if err != nil {
t.Fatalf("SearchFiltered() error = %v", err)
}
if total != searchBatchSize+1 {
t.Fatalf("SearchFiltered() total = %d, want %d", total, searchBatchSize+1)
}
if len(nodes) != searchBatchSize+1 {
t.Fatalf("SearchFiltered() returned %d nodes, want %d", len(nodes), searchBatchSize+1)
}
}
+3
View File
@@ -225,6 +225,9 @@ func Update(ctx context.Context, parent string, objs []model.Obj) {
return
}
unlock := lockUpdate(parent)
defer unlock()
nodes, err := instance.Get(ctx, parent)
if err != nil {
log.Errorf("update search index error while get nodes: %+v", err)
+58
View File
@@ -0,0 +1,58 @@
package search
import (
"testing"
"time"
)
func TestLockUpdateSerializesSameParent(t *testing.T) {
unlockFirst := lockUpdate("/same-parent")
secondStarted := make(chan struct{})
secondAcquired := make(chan struct{})
secondReleased := make(chan struct{})
go func() {
close(secondStarted)
unlockSecond := lockUpdate("/same-parent")
close(secondAcquired)
unlockSecond()
close(secondReleased)
}()
<-secondStarted
select {
case <-secondAcquired:
t.Fatal("second update acquired the same parent lock")
case <-time.After(20 * time.Millisecond):
}
unlockFirst()
select {
case <-secondReleased:
case <-time.After(time.Second):
t.Fatal("second update did not acquire the released parent lock")
}
updateLocksMu.Lock()
defer updateLocksMu.Unlock()
if len(updateLocks) != 0 {
t.Fatalf("update locks were not cleaned up: %d", len(updateLocks))
}
}
func TestLockUpdateAllowsDifferentParents(t *testing.T) {
unlockFirst := lockUpdate("/first-parent")
defer unlockFirst()
secondAcquired := make(chan struct{})
go func() {
unlockSecond := lockUpdate("/second-parent")
unlockSecond()
close(secondAcquired)
}()
select {
case <-secondAcquired:
case <-time.After(time.Second):
t.Fatal("update for a different parent was blocked")
}
}
+40
View File
@@ -51,6 +51,46 @@ func Search(ctx context.Context, req model.SearchReq) ([]model.SearchNode, int64
return instance.Search(ctx, req)
}
const searchBatchSize = 1000
func SearchFiltered(ctx context.Context, req model.SearchReq, filter searcher.Filter) ([]model.SearchNode, int64, error) {
if filteredSearcher, ok := instance.(searcher.FilteredSearcher); ok {
return filteredSearcher.SearchFiltered(ctx, req, filter)
}
batchReq := req
batchReq.Page = 1
batchReq.PerPage = searchBatchSize
from := int64(req.Page-1) * int64(req.PerPage)
to := from + int64(req.PerPage)
var (
result []model.SearchNode
filteredTotal int64
processedTotal int64
)
for {
nodes, total, err := instance.Search(ctx, batchReq)
if err != nil {
return nil, 0, err
}
for _, node := range nodes {
if filter != nil && !filter(node) {
continue
}
if filteredTotal >= from && filteredTotal < to {
result = append(result, node)
}
filteredTotal++
}
processedTotal += int64(len(nodes))
if len(nodes) == 0 || processedTotal >= total {
break
}
batchReq.Page++
}
return result, filteredTotal, nil
}
func Index(ctx context.Context, parent string, obj model.Obj) error {
if instance == nil {
return errs.SearchNotAvailable
+61
View File
@@ -0,0 +1,61 @@
package search
import (
"context"
"testing"
"github.com/OpenListTeam/OpenList/v4/internal/model"
"github.com/OpenListTeam/OpenList/v4/internal/search/searcher"
)
type filteredSearchStub struct {
nodes []model.SearchNode
}
func (s *filteredSearchStub) Config() searcher.Config {
return searcher.Config{Name: "stub"}
}
func (s *filteredSearchStub) Search(_ context.Context, req model.SearchReq) ([]model.SearchNode, int64, error) {
from := (req.Page - 1) * req.PerPage
if from >= len(s.nodes) {
return nil, int64(len(s.nodes)), nil
}
to := min(from+req.PerPage, len(s.nodes))
return s.nodes[from:to], int64(len(s.nodes)), nil
}
func (s *filteredSearchStub) Index(context.Context, model.SearchNode) error { return nil }
func (s *filteredSearchStub) BatchIndex(context.Context, []model.SearchNode) error { return nil }
func (s *filteredSearchStub) Get(context.Context, string) ([]model.SearchNode, error) {
return nil, nil
}
func (s *filteredSearchStub) Del(context.Context, string) error { return nil }
func (s *filteredSearchStub) Release(context.Context) error { return nil }
func (s *filteredSearchStub) Clear(context.Context) error { return nil }
func TestSearchFilteredFiltersBeforePagination(t *testing.T) {
previous := instance
instance = &filteredSearchStub{nodes: []model.SearchNode{
{Name: "denied-1"},
{Name: "allowed-1"},
{Name: "denied-2"},
{Name: "allowed-2"},
}}
t.Cleanup(func() { instance = previous })
nodes, total, err := SearchFiltered(context.Background(), model.SearchReq{
PageReq: model.PageReq{Page: 2, PerPage: 1},
}, func(node model.SearchNode) bool {
return node.Name == "allowed-1" || node.Name == "allowed-2"
})
if err != nil {
t.Fatalf("SearchFiltered() error = %v", err)
}
if total != 2 {
t.Fatalf("SearchFiltered() total = %d, want 2", total)
}
if len(nodes) != 1 || nodes[0].Name != "allowed-2" {
t.Fatalf("SearchFiltered() nodes = %#v, want allowed-2", nodes)
}
}

Some files were not shown because too many files have changed in this diff Show More