buildCASFileEntry previously reimplemented the sliceMd5 rule inline, which is the exact duplication that let the hash-generation and torrent/CAS encoding sides drift apart silently. Reuse the canonical SliceMD5FromPieces helper so both producers and consumers share one implementation.
Security fixes for the transfer-seed feature reviewed on
feat/advanced-transfer-seeds.
SSRF via redirect (torrent.go):
- Source validation only pinned the first hop while http.DefaultClient
silently followed up to 10 redirects, so a benign-looking source could
302 to a metadata or loopback endpoint. Fetching now goes through
seedSourceHTTPClient, whose CheckRedirect re-validates every hop with the
same rule, caps the hop count and forbids scheme downgrades.
- Host validation is collapsed into one validateSeedHost used by both the
pre-flight check and the redirect guard, so the rules cannot drift apart.
- Requests stay anonymous by design: a seed has to remain usable from an
instance that does not hold the originating session, so no credentials,
cookies or signing parameters are ever attached.
Seed content fetching (torrent.go):
- Propagate the request context instead of context.Background(), so
cancellation actually stops the download.
- Stream the body through an io.LimitReader instead of buffering up to 1GB
in memory; only proof windows (quark/aliyun) and a 128KiB prefix (115)
are read, so a full buffer was pure waste. Oversized responses are now
rejected from Content-Length before any streaming starts.
Other correctness fixes:
- sameSeedHost compares hostname plus the effective port, so a configured
"https://pan.example.com" and an embedded "...:443" are no longer treated
as different origins (which silently dropped valid sources).
- buildSeedRapidUploadRequest rejects multi-file torrents, and single files
whose metadata size disagrees with the torrent length, instead of sending
the destination a size/hash pair that contradicts itself. Both call sites
now handle the nil result instead of dereferencing it.
- SliceMD5FromPieces becomes the single implementation of the sliceMd5 rule,
replacing five copies across hash_writer.go, torrent.go, generate.go,
189/torrent.go and 189pc/torrent.go. Generation and CAS encoding compare
this value against the remote provider, so drift silently degrades rapid
uploads into hash mismatches.
- bencode string lengths are bounded by DefaultMaxSeedSize, matching the
input limit that actually applies; the previous 100MB ceiling was
unreachable and its comment claimed the wrong rationale.
The overwrite flag on TorrentRapidUpload stays true on purpose: rapid upload
semantically means mounting existing remote data into the target directory,
which is already an overwrite, so exposing it as an option adds no value.
Tests:
- pkg/torrent/seed_security_test.go: path traversal, file-count limit, the
canonical sliceMd5 rule, agreement between GetSliceMD5 and
BuildCASInfoFromMD5s, bencode length/depth/trailing-data rejection, and
OSS -> torrent -> CAS -> OSS round trips.
- server/handles/torrent_seed_test.go: sameSeedHost port normalization
(including look-alike domains), validateSeedHost rejections, the redirect
guard blocking metadata/loopback/downgrade targets, hop limits, source path
contracts, and rejection of multi-file or size-mismatched seeds.
go build ./... passes; go test ./pkg/torrent/... and
go test ./server/handles/... pass.
The previous commit (624fdd24) introduced the authoritative
driver.SeedRapidUploader interface, but left every driver's seed_rapid.go
on the older, incompatible API, so the branch did not compile at all.
Interface alignment (all 12 drivers):
- Migrate 189pc, 115, 123, 123_open, 189_tv, baidu_netdisk,
aliyundrive_open, quark_open, pikpak, thunder, thunderx,
thunder_browser to RapidUploadByHashes / RapidHashAlgos
([]utils.HashType, no longer []*utils.HashType) / RapidHashNeedsPieces
- Add shared driver.SeedHashStream as a complete model.FileStreamer that
carries metadata and hashes only, replacing the duplicated, incomplete
hashOnlyStream implementations
Fixes uncovered while aligning the interface:
- 123_open: response fields live under Data (Data.Reuse / Data.FileID)
- quark_open: pre.Data.FID -> pre.Data.Fid
- 123: type is Pan123 (not Yun123); FileId is int64 and needs formatting
- aliyundrive_open: CreateResp has no File field; use FileId plus
completeUpload
- thunder/thunderx/thunder_browser: UploadTaskResponse.File is a Files
value type, return &resp.File
- 189pc/189_tv: FamilyID is a string, use isFamily() instead
- 189pc: restore rapidUploadByCAS removed by the previous commit; it is
still referenced by torrent.go. Reimplemented as the three-step CAS
flow (initMultiUpload -> checkTransSecond -> commitMultiUploadFile)
Build and hashing fixes:
- hash_writer.go: HashType exposes NewFunc; GCID.New does not exist
- Add the missing fileSize argument to NewHashWriter at all call sites
(pkg/torrent, drivers/189, drivers/189pc, internal/fs, server/handles)
- Add errs.ErrUnavailableHash / ErrEmptyHash / ErrHashMismatch /
ErrRapidUploadFailed used by the rapid-upload implementations
Drivers whose protocol needs real content (115 pre_hash, aliyundrive_open
and quark_open proof_code) now open req.Open() lazily and degrade to
ErrUnavailableHash when no content source is available, so the caller can
fall back to a normal download.
Note: go vet warnings for non-constant format strings in 189pc/utils.go
are pre-existing and intentionally left untouched.
Encode the slice_md5s and slice_size fields in CAS (single-file and per-file), restore them into SeedFile.Hashes.Pieces.MD5 on decode, and hide the legacy warning when piece hashes are present. Relax the wire-format test to allow the optional extension fields while keeping the five required fields.
CAS now supports multiple files via a files array while keeping the legacy five-field single-file payload byte-compatible. Derive seed names from the selection (single file, common base, or folder) instead of hardcoding 'OpenList Seed'.
Record successful saves as channels and failures as missing_channels when update_channel is set. Return share_status during edit by validating openlist-share sources. Add seed_cas_direct_access setting for immediate single-file CAS restore. Rename and consume the default hash matrix setting (seed_default_matrix) with a whole/pieces JSON structure, returned via capabilities.
Add unified sharing-seed format library (openlist-sharing-seed v1), standard BT torrent v1 with x-openlist/x-cas extensions, and exact legacy-compatible CAS Base64 payload. Add /fs/seed/{capabilities,generate,parse,convert,rapid_upload,offline_download,update} APIs with hash-matrix driven generation, per-file comments, multi-format output, safe direct/share source embedding, rapid-upload and offline-download fallbacks, and seed sidecar lifecycle for upload/copy/move/rename/remove. Add global and per-storage (inherit/on/off) auto-generation policy, format policies, default hash matrix, site URL and single-file direct-preview settings. Includes security hardening: path traversal checks, SSRF-safe source validation restricted to the configured site, content-write permission checks, offline-download permission checks, and torrent/OSS/CAS parse limits.
Add end-to-end offline download support for torrent files, magnet links, and ed2k links, including torrent parse and generate APIs, CAS-based rapid upload for Cloud189, and protocol-aware tool routing with transfer flow improvements.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: j2rong4cn <36783515+j2rong4cn@users.noreply.github.com>
Co-authored-by: j2rong4cn <j2rong@qq.com>
Co-authored-by: Suyunjing <suyunmeng@oplist.org.cn>
* refactor(HybridCache)!: extract hybrid_cache package and rename cache_threshold to auto_memory_limit
* split HybridCache and stores into internal/hybrid_cache package
* introduce BackingStore abstraction with BufferStore and FileStore
* rename CacheThreshold to AutoMemoryLimit in config/env/bootstrap
* update stream/request integration and related tests
* rename singleFileCache and MultiFileCache to singleFileStore and MultiFileStore
* refactor(HybridCache): improve memory management strategies in NewHybridCache
* rename
* alias hybrid_cache to hcache
* omments
* feat(search): enhanced `meilisearch` search experience
- upgrade `meilisearch` dependency
- support subdirectory search
- optimize searchDocument fields for subdirectory search
- specify full index uid instead of index prefix
* fix(search): more fixes to `meilisearch`
- make use of context where context was not used
- remove code of waiting task in deletion process, as tasks are queued and will be executed orderly (if tasks were submitted to the queue successfully), which can improve `AutoUpdate` performance
* fix(archive): unrecognition zip
* feat(archive): add tree for zip meta
* fix bug
* refactor(archive): meta cache time use Link Expiration first
* feat(archive): return sort policy in meta (#2)
* refactor
* perf(archive): reduce new network requests
---------
Co-authored-by: KirCute_ECT <951206789@qq.com>
PIKACHUIM