- C# 99.2%
- Shell 0.6%
- Dockerfile 0.2%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
The release no longer comes from a tag pushed by hand: ci-version.sh gives "stable" the next version of this repository's own tags (YYYY.M.(N+1), or YYYY.M.0 in a new month), the image is pushed and moves "latest", and CI tags the commit after that. A commit that carries a release tag already isn't released again; workflow_dispatch runs a failed release once more. main and every other branch build prerelease images as before, and tags start no build. The tests and the publish are one job now, so the packages are restored once, and the image carries its version and commit as OCI labels. The script is the same file in all three repositories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
| .forgejo/workflows | ||
| fixtures | ||
| scripts | ||
| src | ||
| tests | ||
| .dockerignore | ||
| .gitignore | ||
| CLA.md | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| LICENSE | ||
| README.md | ||
| unilayer.storage.slnx | ||
unilayer storage
unilayer.Storage is the storage service of unilayer: it moves and
transforms the bytes of assets, while the unilayer API keeps their metadata, permissions and versions. Clients get a
signed transfer order from the API and upload or download through this service (or directly from S3).
The service runs next to the unilayer API and uses the same backend (a directory or an S3 bucket), so files uploaded through either are the same files. It also carries out the API's background jobs (optimized copies, cleaning up after a version or a purge) and caches image variants, in memory or in a Valkey shared by all nodes.
Structure
| Project | Content |
|---|---|
src/unilayer.Storage.Contracts |
Token format and signing, transfer orders, job contracts, the optimized copies' names - what the unilayer API references |
src/unilayer.Storage.Abstractions |
Extension points (ITenantResolver, IKeyProvider, ICorsPolicyProvider, IUsageReporter) |
src/unilayer.Storage |
Providers (local, S3), token validation, transformations, cache, jobs, endpoints |
src/unilayer.Storage.Host |
The service and its Docker image unilayer-storage |
The service runs as a single node (local or S3 backend, the instance's Valkey) or as several nodes for a unilayer cluster (S3 only, a Valkey of its own).
Running it
dotnet run --project src/unilayer.Storage.Host -- --storage:keys:url=http://localhost:5380/api/v1/storage/keys
Or the image code.talaryon.dev/unilayer/unilayer-storage (port 8080), built from a publish/ folder that
dotnet publish src/unilayer.Storage.Host -o publish made before (the CI does). Settings come from
appsettings.json, UNILAYER_* environment variables (UNILAYER_S3_BUCKETNAME is s3:bucketName) and command line
arguments:
| Setting | Environment | Default | Description |
|---|---|---|---|
storage:backend |
UNILAYER_STORAGE_BACKEND |
local |
local (a directory, one node only) or s3 (also minio). storage:provider is read too, as before |
storage:mode |
UNILAYER_STORAGE_MODE |
single |
single or cluster: several nodes behind a load balancer. A cluster node needs s3, storage:valkey:configuration (a writable Valkey of the service's own) and storage:results:signing:privateKey (the same on every node), and refuses to start without them |
local:path |
UNILAYER_LOCAL_PATH |
./storage/assets, in the image /data/assets |
The directory; the same as the server's. The service refuses to start when it can't write to it |
s3:endpoint, s3:accessKey, s3:secretKey, s3:bucketName, s3:basePath, s3:region |
UNILAYER_S3_* |
bucket unilayer-assets, base path assets |
The bucket; the same as the server's |
s3:useSsl |
UNILAYER_S3_USESSL |
unset | HTTPS for an endpoint without a scheme. Unset, it is on unless the store is local (a loopback or private address, or a name without a dot like minio), as in the server; a scheme in the endpoint wins. Explicitly false for a store beyond the local network is warned about |
s3:createBucketIfNotExists |
UNILAYER_S3_CREATEBUCKETIFNOTEXISTS |
true |
Creates the bucket before the first write when it isn't there; a key that may only use the bucket is fine |
storage:issuer |
UNILAYER_STORAGE_ISSUER |
unilayer |
The instance's issuer name (its storage:signing:issuer) |
storage:prefix |
UNILAYER_STORAGE_PREFIX |
empty | The instance's folder in the backend |
storage:audience |
UNILAYER_STORAGE_AUDIENCE |
none | The service's own name: tokens and jobs whose aud names another service are refused, those without aud taken |
storage:keys:url |
UNILAYER_STORAGE_KEYS_URL |
The instance's public keys, GET /api/v1/storage/keys of the API |
|
storage:keys:jwks |
UNILAYER_STORAGE_KEYS_JWKS |
The key set itself, instead of the URL | |
storage:keys:refreshMinutes |
UNILAYER_STORAGE_KEYS_REFRESHMINUTES |
5 |
How often the key set at the URL is read again; an unknown key id reads it at once, at most every 30 s |
storage:tokens:maxLifetimeHours |
UNILAYER_STORAGE_TOKENS_MAXLIFETIMEHOURS |
168 |
A token expiring later than this from now is refused; 0 for no limit |
storage:cors:origins |
UNILAYER_STORAGE_CORS_ORIGINS |
none | Comma-separated web origins allowed to call it; * for any |
storage:maxUploadBytes |
UNILAYER_STORAGE_MAXUPLOADBYTES |
10 GiB | The largest upload, whatever a token allows |
storage:snapshots:maxBytes |
UNILAYER_STORAGE_SNAPSHOTS_MAXBYTES |
50 GiB | The largest cluster snapshot, whatever a token allows; storage:maxUploadBytes doesn't apply to snapshots |
storage:transform:* |
UNILAYER_STORAGE_TRANSFORM_* |
Image modifier limits, see below; the server's assets:transform:* are read too |
|
storage:optimize:minSavingsPercent |
UNILAYER_STORAGE_OPTIMIZE_MINSAVINGSPERCENT |
5 |
An optimized copy is kept only when it is this much smaller; the server's assets:optimize:* is read too |
storage:instance:url |
UNILAYER_STORAGE_INSTANCE_URL |
the address of storage:keys:url |
The API, where job results go (POST /api/v1/storage/job-results) |
storage:results:signing:privateKey, …:privateKeyFile, …:keyId |
UNILAYER_STORAGE_RESULTS_SIGNING_* |
a key made at every start, storage-1 |
The ES256 key job results are signed with, published in GET /info. The same on every node of a cluster |
storage:valkey:configuration |
UNILAYER_STORAGE_VALKEY_CONFIGURATION |
none | A Valkey (StackExchange.Redis configuration, e.g. valkey:6379,user=storage,password=...) for the job queue, the variant cache and its locks, shared by all nodes; without it each node keeps them in memory |
storage:valkey:prefix |
UNILAYER_STORAGE_VALKEY_PREFIX |
unilayer-storage: |
Put before every key, so the service can share the instance's Valkey |
storage:jobs:concurrency |
UNILAYER_STORAGE_JOBS_CONCURRENCY |
1 |
Jobs carried out at once on a node |
storage:jobs:maxAgeDays |
UNILAYER_STORAGE_JOBS_MAXAGEDAYS |
0 |
A job made longer ago is refused; 0 for any age. The instance sends a job again with its first createdAt, so a limit refuses jobs that waited that long |
The image modifier limits (storage:transform:*): maxDimension (4096 px), maxSourcePixels (50,000,000, all
frames together), maxSourceBytes (100 MiB, also for optimized copies), maxScale (4), defaultQuality (80),
maxConcurrentTransforms (0: one per memoryPerTransformBytes, 256 MiB, of the memory the process may use, at most
one per core), maxPixelMemoryBytes (0: a quarter of the process's memory), maxImageSeconds (120: how long
ImageMagick may work on one image), queueTimeoutSeconds (10), cacheTtlMinutes (60), maxCachedBytes (1 MiB),
maxStoredVariants (50) and maxRendersPerFileMinute (30: new variants of one file per node and minute, 429 with
Retry-After beyond; 0 for no limit).
Don't share the environment with the server. Some storage:* names mean something else there: storage:provider
is service, and storage:signing:* is the instance's own private key, which the service must not hold. The service
names its own storage:backend and storage:results:signing:*; it still reads storage:signing:* under the old
name with a warning, but never one that names an issuer or a standby key - that is the server's.
Jobs and the variant cache
The instance hands the service work to do in the background: optimize an image, remove what an older
version left, remove a purged asset's files. POST /v1/jobs takes a JSON array of jobs, each signed with
the instance's key (storage-job+jwt), and answers 202 with how many it queued and why it refused the
others (StorageJobsAccepted). A node carries out each job once and reports the result, signed with the
service's key, to the instance - apart from the work, up to 100 results in one request. The instance keeps
every job until it has a result and sends it again otherwise, so the queue - in Valkey a stream with one
consumer group and about 100,000 jobs at most, without Valkey in memory - may lose them, and every job can
run twice without harm. An idle node asks the queue every 0.5 to 5 s, and at once when a job is added.
The optimizer takes JPEG, PNG and GIF up to maxSourceBytes, checked before the file is loaded; what it skips, and
why, is stored in the version's descriptor too, so it isn't tried again.
Variants made with image modifiers are cached per file version and modifiers: up to maxCachedBytes in Valkey (or
in memory), larger ones as files under variants/{uid}/v{version}/ with Valkey holding where, at most
maxStoredVariants of a version; they stay until the version is cleaned up. Requests on a node for the same variant
share one render; while another node renders it (a SET NX lock in Valkey), they wait for it. Without Valkey the
cache is in memory and per node. Valkey is for efficiency only: what it loses is redone. Give it a maxmemory with
volatile-lru: the cached variants expire and may be evicted, the job stream has no expiry and must not be. Every
combination of modifiers in a public image URL is a variant of its own: a node renders at most
maxRendersPerFileMinute new ones of a file per minute, so one URL can't be walked through every size. Limit requests
per client in front of the service (ingress, CDN) as well if that matters.
HTTP API
Every file request carries a token from the API, as ?token= or Authorization: Bearer. It names the path, the
operation and the limits.
| Request | Token | Does |
|---|---|---|
GET / HEAD /v1/files/{path} |
read or transform |
Downloads the file, also a Range of it; ?disposition=inline shows it in the browser, but only raster images, video, audio, PDF and plain text (by the token's ct): SVG, HTML and anything else stay a download, with Content-Security-Policy: sandbox. With a transform token, width, height, scale, fit, format and quality as in the API |
PUT /v1/files/{path} |
write |
Uploads the file in one request; it takes its place only when its SHA-256 is the announced one (422 otherwise, 413 when too large) |
PUT /v1/files/{path}?part=n |
write |
Uploads part n of a multipart upload; the response's ETag names the part. The parts stored together stay within the token's max (413) |
GET /v1/files/{path}?parts |
write |
The parts stored so far as [{"partNumber":1,"etag":"..."}], for a client that resumes an upload; read from the SHA-256 stored next to each part, not from the parts |
POST /v1/files/{path} |
write |
Puts the parts together: a JSON array of {"partNumber":1,"etag":"..."}, numbered from 1. In S3 the store copies them together itself when all but the last have 5 MB, and the service reads the result once to check it; elsewhere it writes them together as it reads |
DELETE /v1/files/{path} |
delete |
Deletes the file; 404 when there is none. For the instance itself |
POST /v1/move/{path} |
move |
Moves the file to the token's to, replacing what is there; 404 when there is none. For the instance itself |
GET /v1/list/{folder} |
list |
The files under the folder and its subfolders as a JSON array of {"path","size","modified"}, paths relative to the instance's prefix; GET /v1/list with an empty token path for all of them. For the instance itself |
PUT /v1/snapshots/{name} |
snapshot:write |
Uploads a cluster snapshot, streamed; it takes its place only when its SHA-256 is the announced one (422 otherwise, 413 when too large) |
GET / HEAD /v1/snapshots/{name} |
snapshot:read |
Downloads a cluster snapshot, streamed |
GET /v1/snapshots |
snapshot:list |
The instance's snapshots as a JSON array of {"name","size","modified"} |
DELETE /v1/snapshots/{name} |
snapshot:delete |
Deletes a snapshot; 404 when there is none |
POST /v1/jobs |
none, each job is signed | Queues jobs of an instance (see above). Only the instance sends them: it needn't be reachable from the internet |
GET /info |
none | The contract versions the service reads, the keys of its job results, its license and source |
GET / |
none | The service, its version, license and source |
GET /health, /health/live |
none | The process answers |
GET /health/ready |
none | It has the instance's keys and reaches its backend (degraded without Valkey) |
A request body that isn't an upload (jobs, the parts to put together) is at most 4 MiB; an upload may be as large as
its token's max, a snapshot's at most storage:snapshots:maxBytes. The parts of an upload that is never completed stay under uploads/, where the instance removes
them. Files are sent with X-Content-Type-Options: nosniff and Referrer-Policy: no-referrer, so a page that embeds a
download link doesn't pass its token on.
Errors are {"code":"...","message":"..."} (StorageError); clients go by the code (StorageErrorCodes), the
status only says the kind of error.
Cluster snapshots
A unilayer cluster keeps its Raft snapshots - a pg_dump of the database, up to many GB - here, so its nodes need no
S3 key of their own. They are the files of {prefix}/cluster-snapshots/ and reached only through /v1/snapshots, with
a token of their own type (storage-snapshot+jwt, SnapshotToken) that the instance signs with its storage key for
each request: op is snapshot:write, snapshot:read, snapshot:list or snapshot:delete, name the snapshot
(all but a listing), max and sha the upload's limit and SHA-256, and it is valid for an hour at most. A name is
[0-9a-z-]+.dump (unilayer's are {index}-{sha12}.dump), so nothing leaves the folder; an asset token naming a path
in it is refused. An upload is written next to its place and moved there once its SHA-256 checked out. The instance
prunes old snapshots itself (list and delete). Their traffic is reported in StorageUsage.SnapshotBytesIn and
SnapshotBytesOut, not in BytesIn and BytesOut, so it isn't counted as the assets'.
Security
- The key set says who may sign tokens and jobs for the instance, delete, move and list included: read it over
HTTPS, or give it as
storage:keys:jwksin production. Keys read over plain HTTP from beyond the local network are warned about. - Tokens travel in URLs. The service logs only paths; keep the ASP.NET Core hosting log (
Logging:LogLevel:Microsoft.AspNetCore) atWarning, asappsettings.jsonhas it - atInformationit logs every URL with its?token=. - Images are decoded only as JPEG, PNG, GIF, WebP, BMP or TIFF, detected from their bytes, with ImageMagick's coder forced to that format: SVG, HEIC, AVIF, PDF and video are never decoded. The native Magick library still contains codecs such as libheif, libde265 (HEVC), aom and openh264 (H.264); HEVC and H.264 are patented, which matters to whoever distributes the image commercially, not to what the service does with them.
Kubernetes
The image runs as user 1654 (runAsNonRoot can check it) and listens on 8080. Probe /health/live for liveness and
/health/ready for readiness. With the local backend mount a volume at /data/assets; with
readOnlyRootFilesystem also an emptyDir at /tmp, where ImageMagick and the optimizer write their temporary files.
Contracts
unilayer.Storage.Contracts has no dependencies beyond .NET.
- Tokens (
Tokens/): JWTs signed with ES256 (ECDSA on P-256) by the unilayer instance that issues them, header{"alg":"ES256","typ":"JWT","kid":...}. The claims arev(token format),iss(the instance),uid,ver,path(relative to the instance's prefix),op(read,write,transform,delete,move,list),maxandsha(writes only),ctandname(the type and file name a download is sent with),to(moves only),aud(the service, optional) andexp. Validation accepts ES256 only, needs the key named bykid, and accepts a token only when that key belongs to itsiss. Public keys travel as JSON Web Keys (StorageJsonWebKeySet). - Transfer orders (
Transfers/): what a client carries out to move a file -url,method, the signedheaders,expiresAt, andpartsplusuploadIdfor a multipart upload. - Jobs (
Jobs/): the work the service does for an instance (optimize,variants.cleanUp,files.delete), the answer toPOST /v1/jobs(StorageJobsAccepted) and the results it reports, signed with the service's own key. - Optimized copies (
Optimization/): where the optimized copy of a version and its descriptor are stored, and the descriptor itself. StorageInfo: whatGET /infoanswers - the contract versions the service reads, its keys, license and source.StorageFileEntry: a file in the answer toGET /v1/list.- Snapshot tokens (
Tokens/SnapshotToken,SnapshotTokenClaims) andSnapshotEntry, a snapshot in the answer toGET /v1/snapshots.StorageInfo.Contracts.Snapshotssays which snapshot token formats a service reads; a service without it keeps no snapshots. StorageError: the body of every error response.
ES256 signatures are randomized: the same claims give the same header and payload but another signature. For a
download URL that stays the same within a time window (StorageToken.WindowedExpiry), the instance keeps the token
it signed for the window and hands it out again.
The JSON examples in fixtures/ are the contracts as other repositories see them. The tests here read
every one into its type and write it back unchanged; the client tests its own copies of the transfer orders against
the same files.
dotnet test unilayer.storage.slnx
The S3 and Valkey tests run against a store and a Valkey given in UNILAYER_TEST_S3_ENDPOINT, _ACCESSKEY,
_SECRETKEY (and _BUCKET) and UNILAYER_TEST_VALKEY, e.g. localhost:6379; without them they are skipped. The CI
starts both as services.
Rules
- This repository references neither the unilayer server nor the client. The server includes it as a submodule
(
libs/storage) and referencesunilayer.Storage.Contractsonly. - Tokens and job messages carry a contract version; the service accepts at least the current and the previous one.
- The optimizer,
JpegSegmentsand the image transforms have a copy in the server; until the server's is gone, a fix goes into both.
Releases
A push to stable releases the next YYYY.M.N (CalVer, no leading zeros; N counts this repository's releases within
a month, from 0) and CI tags it once the image is pushed. Every other push builds a prerelease of the next version.
License
AGPL-3.0-only. Contributions require the Contributor License Agreement, see
CONTRIBUTING. The image carries the license and the notices of the libraries it contains in
/app/licenses.