No description
  • C# 99.2%
  • Shell 0.6%
  • Dockerfile 0.2%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
ferociousbyte 1a3e8fe7d2
All checks were successful
/ build (push) Successful in 1m4s
/ push (push) Successful in 46s
A push to stable releases the next YYYY.M.N and tags it, other pushes build CI images only
The release no longer comes from a tag pushed by hand: ci-version.sh gives "stable" the next version of this
repository's own tags (YYYY.M.(N+1), or YYYY.M.0 in a new month), the image is pushed and moves "latest", and CI tags
the commit after that. A commit that carries a release tag already isn't released again; workflow_dispatch runs a
failed release once more. main and every other branch build prerelease images as before, and tags start no build.

The tests and the publish are one job now, so the packages are restored once, and the image carries its version and
commit as OCI labels. The script is the same file in all three repositories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 11:59:10 +01:00
.forgejo/workflows A push to stable releases the next YYYY.M.N and tags it, other pushes build CI images only 2026-10-25 11:59:10 +01:00
fixtures The service keeps an instance's cluster snapshots under /v1/snapshots 2026-10-25 10:55:31 +01:00
scripts A push to stable releases the next YYYY.M.N and tags it, other pushes build CI images only 2026-10-25 11:59:10 +01:00
src The service keeps an instance's cluster snapshots under /v1/snapshots 2026-10-25 10:55:31 +01:00
tests The service keeps an instance's cluster snapshots under /v1/snapshots 2026-10-25 10:55:31 +01:00
.dockerignore Add the storage service: core, extension points and host 2026-10-23 16:21:48 +02:00
.gitignore Set up the storage repository 2026-10-23 12:48:06 +02:00
CLA.md Set up the storage repository 2026-10-23 12:48:06 +02:00
CONTRIBUTING.md Set up the storage repository 2026-10-23 12:48:06 +02:00
Dockerfile The image carries its licenses and third-party notices, and the service offers its source 2026-10-24 23:49:31 +02:00
LICENSE Set up the storage repository 2026-10-23 12:48:06 +02:00
README.md A push to stable releases the next YYYY.M.N and tags it, other pushes build CI images only 2026-10-25 11:59:10 +01:00
unilayer.storage.slnx Add the storage service: core, extension points and host 2026-10-23 16:21:48 +02:00

unilayer storage

unilayer.Storage is the storage service of unilayer: it moves and transforms the bytes of assets, while the unilayer API keeps their metadata, permissions and versions. Clients get a signed transfer order from the API and upload or download through this service (or directly from S3).

The service runs next to the unilayer API and uses the same backend (a directory or an S3 bucket), so files uploaded through either are the same files. It also carries out the API's background jobs (optimized copies, cleaning up after a version or a purge) and caches image variants, in memory or in a Valkey shared by all nodes.

Structure

Project Content
src/unilayer.Storage.Contracts Token format and signing, transfer orders, job contracts, the optimized copies' names - what the unilayer API references
src/unilayer.Storage.Abstractions Extension points (ITenantResolver, IKeyProvider, ICorsPolicyProvider, IUsageReporter)
src/unilayer.Storage Providers (local, S3), token validation, transformations, cache, jobs, endpoints
src/unilayer.Storage.Host The service and its Docker image unilayer-storage

The service runs as a single node (local or S3 backend, the instance's Valkey) or as several nodes for a unilayer cluster (S3 only, a Valkey of its own).

Running it

dotnet run --project src/unilayer.Storage.Host -- --storage:keys:url=http://localhost:5380/api/v1/storage/keys

Or the image code.talaryon.dev/unilayer/unilayer-storage (port 8080), built from a publish/ folder that dotnet publish src/unilayer.Storage.Host -o publish made before (the CI does). Settings come from appsettings.json, UNILAYER_* environment variables (UNILAYER_S3_BUCKETNAME is s3:bucketName) and command line arguments:

Setting Environment Default Description
storage:backend UNILAYER_STORAGE_BACKEND local local (a directory, one node only) or s3 (also minio). storage:provider is read too, as before
storage:mode UNILAYER_STORAGE_MODE single single or cluster: several nodes behind a load balancer. A cluster node needs s3, storage:valkey:configuration (a writable Valkey of the service's own) and storage:results:signing:privateKey (the same on every node), and refuses to start without them
local:path UNILAYER_LOCAL_PATH ./storage/assets, in the image /data/assets The directory; the same as the server's. The service refuses to start when it can't write to it
s3:endpoint, s3:accessKey, s3:secretKey, s3:bucketName, s3:basePath, s3:region UNILAYER_S3_* bucket unilayer-assets, base path assets The bucket; the same as the server's
s3:useSsl UNILAYER_S3_USESSL unset HTTPS for an endpoint without a scheme. Unset, it is on unless the store is local (a loopback or private address, or a name without a dot like minio), as in the server; a scheme in the endpoint wins. Explicitly false for a store beyond the local network is warned about
s3:createBucketIfNotExists UNILAYER_S3_CREATEBUCKETIFNOTEXISTS true Creates the bucket before the first write when it isn't there; a key that may only use the bucket is fine
storage:issuer UNILAYER_STORAGE_ISSUER unilayer The instance's issuer name (its storage:signing:issuer)
storage:prefix UNILAYER_STORAGE_PREFIX empty The instance's folder in the backend
storage:audience UNILAYER_STORAGE_AUDIENCE none The service's own name: tokens and jobs whose aud names another service are refused, those without aud taken
storage:keys:url UNILAYER_STORAGE_KEYS_URL The instance's public keys, GET /api/v1/storage/keys of the API
storage:keys:jwks UNILAYER_STORAGE_KEYS_JWKS The key set itself, instead of the URL
storage:keys:refreshMinutes UNILAYER_STORAGE_KEYS_REFRESHMINUTES 5 How often the key set at the URL is read again; an unknown key id reads it at once, at most every 30 s
storage:tokens:maxLifetimeHours UNILAYER_STORAGE_TOKENS_MAXLIFETIMEHOURS 168 A token expiring later than this from now is refused; 0 for no limit
storage:cors:origins UNILAYER_STORAGE_CORS_ORIGINS none Comma-separated web origins allowed to call it; * for any
storage:maxUploadBytes UNILAYER_STORAGE_MAXUPLOADBYTES 10 GiB The largest upload, whatever a token allows
storage:snapshots:maxBytes UNILAYER_STORAGE_SNAPSHOTS_MAXBYTES 50 GiB The largest cluster snapshot, whatever a token allows; storage:maxUploadBytes doesn't apply to snapshots
storage:transform:* UNILAYER_STORAGE_TRANSFORM_* Image modifier limits, see below; the server's assets:transform:* are read too
storage:optimize:minSavingsPercent UNILAYER_STORAGE_OPTIMIZE_MINSAVINGSPERCENT 5 An optimized copy is kept only when it is this much smaller; the server's assets:optimize:* is read too
storage:instance:url UNILAYER_STORAGE_INSTANCE_URL the address of storage:keys:url The API, where job results go (POST /api/v1/storage/job-results)
storage:results:signing:privateKey, …:privateKeyFile, …:keyId UNILAYER_STORAGE_RESULTS_SIGNING_* a key made at every start, storage-1 The ES256 key job results are signed with, published in GET /info. The same on every node of a cluster
storage:valkey:configuration UNILAYER_STORAGE_VALKEY_CONFIGURATION none A Valkey (StackExchange.Redis configuration, e.g. valkey:6379,user=storage,password=...) for the job queue, the variant cache and its locks, shared by all nodes; without it each node keeps them in memory
storage:valkey:prefix UNILAYER_STORAGE_VALKEY_PREFIX unilayer-storage: Put before every key, so the service can share the instance's Valkey
storage:jobs:concurrency UNILAYER_STORAGE_JOBS_CONCURRENCY 1 Jobs carried out at once on a node
storage:jobs:maxAgeDays UNILAYER_STORAGE_JOBS_MAXAGEDAYS 0 A job made longer ago is refused; 0 for any age. The instance sends a job again with its first createdAt, so a limit refuses jobs that waited that long

The image modifier limits (storage:transform:*): maxDimension (4096 px), maxSourcePixels (50,000,000, all frames together), maxSourceBytes (100 MiB, also for optimized copies), maxScale (4), defaultQuality (80), maxConcurrentTransforms (0: one per memoryPerTransformBytes, 256 MiB, of the memory the process may use, at most one per core), maxPixelMemoryBytes (0: a quarter of the process's memory), maxImageSeconds (120: how long ImageMagick may work on one image), queueTimeoutSeconds (10), cacheTtlMinutes (60), maxCachedBytes (1 MiB), maxStoredVariants (50) and maxRendersPerFileMinute (30: new variants of one file per node and minute, 429 with Retry-After beyond; 0 for no limit).

Don't share the environment with the server. Some storage:* names mean something else there: storage:provider is service, and storage:signing:* is the instance's own private key, which the service must not hold. The service names its own storage:backend and storage:results:signing:*; it still reads storage:signing:* under the old name with a warning, but never one that names an issuer or a standby key - that is the server's.

Jobs and the variant cache

The instance hands the service work to do in the background: optimize an image, remove what an older version left, remove a purged asset's files. POST /v1/jobs takes a JSON array of jobs, each signed with the instance's key (storage-job+jwt), and answers 202 with how many it queued and why it refused the others (StorageJobsAccepted). A node carries out each job once and reports the result, signed with the service's key, to the instance - apart from the work, up to 100 results in one request. The instance keeps every job until it has a result and sends it again otherwise, so the queue - in Valkey a stream with one consumer group and about 100,000 jobs at most, without Valkey in memory - may lose them, and every job can run twice without harm. An idle node asks the queue every 0.5 to 5 s, and at once when a job is added.

The optimizer takes JPEG, PNG and GIF up to maxSourceBytes, checked before the file is loaded; what it skips, and why, is stored in the version's descriptor too, so it isn't tried again.

Variants made with image modifiers are cached per file version and modifiers: up to maxCachedBytes in Valkey (or in memory), larger ones as files under variants/{uid}/v{version}/ with Valkey holding where, at most maxStoredVariants of a version; they stay until the version is cleaned up. Requests on a node for the same variant share one render; while another node renders it (a SET NX lock in Valkey), they wait for it. Without Valkey the cache is in memory and per node. Valkey is for efficiency only: what it loses is redone. Give it a maxmemory with volatile-lru: the cached variants expire and may be evicted, the job stream has no expiry and must not be. Every combination of modifiers in a public image URL is a variant of its own: a node renders at most maxRendersPerFileMinute new ones of a file per minute, so one URL can't be walked through every size. Limit requests per client in front of the service (ingress, CDN) as well if that matters.

HTTP API

Every file request carries a token from the API, as ?token= or Authorization: Bearer. It names the path, the operation and the limits.

Request Token Does
GET / HEAD /v1/files/{path} read or transform Downloads the file, also a Range of it; ?disposition=inline shows it in the browser, but only raster images, video, audio, PDF and plain text (by the token's ct): SVG, HTML and anything else stay a download, with Content-Security-Policy: sandbox. With a transform token, width, height, scale, fit, format and quality as in the API
PUT /v1/files/{path} write Uploads the file in one request; it takes its place only when its SHA-256 is the announced one (422 otherwise, 413 when too large)
PUT /v1/files/{path}?part=n write Uploads part n of a multipart upload; the response's ETag names the part. The parts stored together stay within the token's max (413)
GET /v1/files/{path}?parts write The parts stored so far as [{"partNumber":1,"etag":"..."}], for a client that resumes an upload; read from the SHA-256 stored next to each part, not from the parts
POST /v1/files/{path} write Puts the parts together: a JSON array of {"partNumber":1,"etag":"..."}, numbered from 1. In S3 the store copies them together itself when all but the last have 5 MB, and the service reads the result once to check it; elsewhere it writes them together as it reads
DELETE /v1/files/{path} delete Deletes the file; 404 when there is none. For the instance itself
POST /v1/move/{path} move Moves the file to the token's to, replacing what is there; 404 when there is none. For the instance itself
GET /v1/list/{folder} list The files under the folder and its subfolders as a JSON array of {"path","size","modified"}, paths relative to the instance's prefix; GET /v1/list with an empty token path for all of them. For the instance itself
PUT /v1/snapshots/{name} snapshot:write Uploads a cluster snapshot, streamed; it takes its place only when its SHA-256 is the announced one (422 otherwise, 413 when too large)
GET / HEAD /v1/snapshots/{name} snapshot:read Downloads a cluster snapshot, streamed
GET /v1/snapshots snapshot:list The instance's snapshots as a JSON array of {"name","size","modified"}
DELETE /v1/snapshots/{name} snapshot:delete Deletes a snapshot; 404 when there is none
POST /v1/jobs none, each job is signed Queues jobs of an instance (see above). Only the instance sends them: it needn't be reachable from the internet
GET /info none The contract versions the service reads, the keys of its job results, its license and source
GET / none The service, its version, license and source
GET /health, /health/live none The process answers
GET /health/ready none It has the instance's keys and reaches its backend (degraded without Valkey)

A request body that isn't an upload (jobs, the parts to put together) is at most 4 MiB; an upload may be as large as its token's max, a snapshot's at most storage:snapshots:maxBytes. The parts of an upload that is never completed stay under uploads/, where the instance removes them. Files are sent with X-Content-Type-Options: nosniff and Referrer-Policy: no-referrer, so a page that embeds a download link doesn't pass its token on.

Errors are {"code":"...","message":"..."} (StorageError); clients go by the code (StorageErrorCodes), the status only says the kind of error.

Cluster snapshots

A unilayer cluster keeps its Raft snapshots - a pg_dump of the database, up to many GB - here, so its nodes need no S3 key of their own. They are the files of {prefix}/cluster-snapshots/ and reached only through /v1/snapshots, with a token of their own type (storage-snapshot+jwt, SnapshotToken) that the instance signs with its storage key for each request: op is snapshot:write, snapshot:read, snapshot:list or snapshot:delete, name the snapshot (all but a listing), max and sha the upload's limit and SHA-256, and it is valid for an hour at most. A name is [0-9a-z-]+.dump (unilayer's are {index}-{sha12}.dump), so nothing leaves the folder; an asset token naming a path in it is refused. An upload is written next to its place and moved there once its SHA-256 checked out. The instance prunes old snapshots itself (list and delete). Their traffic is reported in StorageUsage.SnapshotBytesIn and SnapshotBytesOut, not in BytesIn and BytesOut, so it isn't counted as the assets'.

Security

  • The key set says who may sign tokens and jobs for the instance, delete, move and list included: read it over HTTPS, or give it as storage:keys:jwks in production. Keys read over plain HTTP from beyond the local network are warned about.
  • Tokens travel in URLs. The service logs only paths; keep the ASP.NET Core hosting log (Logging:LogLevel:Microsoft.AspNetCore) at Warning, as appsettings.json has it - at Information it logs every URL with its ?token=.
  • Images are decoded only as JPEG, PNG, GIF, WebP, BMP or TIFF, detected from their bytes, with ImageMagick's coder forced to that format: SVG, HEIC, AVIF, PDF and video are never decoded. The native Magick library still contains codecs such as libheif, libde265 (HEVC), aom and openh264 (H.264); HEVC and H.264 are patented, which matters to whoever distributes the image commercially, not to what the service does with them.

Kubernetes

The image runs as user 1654 (runAsNonRoot can check it) and listens on 8080. Probe /health/live for liveness and /health/ready for readiness. With the local backend mount a volume at /data/assets; with readOnlyRootFilesystem also an emptyDir at /tmp, where ImageMagick and the optimizer write their temporary files.

Contracts

unilayer.Storage.Contracts has no dependencies beyond .NET.

  • Tokens (Tokens/): JWTs signed with ES256 (ECDSA on P-256) by the unilayer instance that issues them, header {"alg":"ES256","typ":"JWT","kid":...}. The claims are v (token format), iss (the instance), uid, ver, path (relative to the instance's prefix), op (read, write, transform, delete, move, list), max and sha (writes only), ct and name (the type and file name a download is sent with), to (moves only), aud (the service, optional) and exp. Validation accepts ES256 only, needs the key named by kid, and accepts a token only when that key belongs to its iss. Public keys travel as JSON Web Keys (StorageJsonWebKeySet).
  • Transfer orders (Transfers/): what a client carries out to move a file - url, method, the signed headers, expiresAt, and parts plus uploadId for a multipart upload.
  • Jobs (Jobs/): the work the service does for an instance (optimize, variants.cleanUp, files.delete), the answer to POST /v1/jobs (StorageJobsAccepted) and the results it reports, signed with the service's own key.
  • Optimized copies (Optimization/): where the optimized copy of a version and its descriptor are stored, and the descriptor itself.
  • StorageInfo: what GET /info answers - the contract versions the service reads, its keys, license and source.
  • StorageFileEntry: a file in the answer to GET /v1/list.
  • Snapshot tokens (Tokens/SnapshotToken, SnapshotTokenClaims) and SnapshotEntry, a snapshot in the answer to GET /v1/snapshots. StorageInfo.Contracts.Snapshots says which snapshot token formats a service reads; a service without it keeps no snapshots.
  • StorageError: the body of every error response.

ES256 signatures are randomized: the same claims give the same header and payload but another signature. For a download URL that stays the same within a time window (StorageToken.WindowedExpiry), the instance keeps the token it signed for the window and hands it out again.

The JSON examples in fixtures/ are the contracts as other repositories see them. The tests here read every one into its type and write it back unchanged; the client tests its own copies of the transfer orders against the same files.

dotnet test unilayer.storage.slnx

The S3 and Valkey tests run against a store and a Valkey given in UNILAYER_TEST_S3_ENDPOINT, _ACCESSKEY, _SECRETKEY (and _BUCKET) and UNILAYER_TEST_VALKEY, e.g. localhost:6379; without them they are skipped. The CI starts both as services.

Rules

  • This repository references neither the unilayer server nor the client. The server includes it as a submodule (libs/storage) and references unilayer.Storage.Contracts only.
  • Tokens and job messages carry a contract version; the service accepts at least the current and the previous one.
  • The optimizer, JpegSegments and the image transforms have a copy in the server; until the server's is gone, a fix goes into both.

Releases

A push to stable releases the next YYYY.M.N (CalVer, no leading zeros; N counts this repository's releases within a month, from 0) and CI tags it once the image is pushed. Every other push builds a prerelease of the next version.

License

AGPL-3.0-only. Contributions require the Contributor License Agreement, see CONTRIBUTING. The image carries the license and the notices of the libraries it contains in /app/licenses.