Seed packs and declarative tenant seeding

Seed packs can be restricted to specific realms using includeRealms or excludeRealms in the scope block of the manifest. This controls which realms a seed pack applies to during provisioning.

These two fields are mutually exclusive — setting both causes a validation error at manifest load time.

Realm filtering composes with the scope type. The type-based check (GLOBAL, PER_TENANT, TENANT_LIST, etc.) runs first; if it passes, the realm filter is applied as an additional constraint.

Callers can also use loader.applyArchetype("FulfillmentPlus", ctx) to resolve an archetype defined in any manifest.

The framework now applies seed packs via a dedicated SeedStartupRunner, independent of schema migrations. This runner discovers and applies the latest version of each seed pack for important realms (system/default/test) on application startup.

Key points: - Discovery: SeedStartupRunner constructs a SeedLoader with a FileSeedSource pointing at the configured seed root. Set quantum.seed.root (for tests we default to src/main/resources/seed-packs). The source walks the directory tree and locates every manifest.yaml file. - Selection: For each discovered seed pack name, the runner selects the latest semantic version and builds SeedPackRef.exact(name, version) for application. - Execution: The runner builds a SeedContext for the target realm and calls loader.apply(refs, context). Indexes declared in the manifest are created before data is upserted. - Idempotency + repeatable: The MongoSeedRegistry stores a checksum per dataset in the realm’s _seed_registry collection. If the checksum matches on a later run, the dataset is skipped; if it changes, the dataset is re-applied. - Concurrency safety: The runner uses a Sherlock distributed lock per realm to prevent concurrent execution across nodes.

Configuration snippet:

Transforms are small, composable functions that shape each dataset record just before it is written to the database. They let you keep dataset files generic and inject environment/tenant specifics or perform repeatable rewrites at apply time.

What a transform gets and returns: - Input: the current record (a Map), the SeedContext, and the Dataset definition - Output: the next record (Map) to be passed to the rest of the pipeline; return null or an empty map to drop the record

Where transforms are declared (manifest):

Execution semantics: - Ordering: transforms are executed top-to-bottom for each record - Short-circuit: if any transform returns null or an empty map, the record is skipped and no write occurs - Overwrite rules: a transform can set or overwrite fields on the record; when upsert=true, the final transformed record replaces the existing one matched by naturalKey - Interaction with naturalKey and indexes: transforms run before naturalKey validation and index creation; do not remove fields listed in naturalKey, otherwise an error will be thrown during write

Built-in transform types:

Built-in variables available from SeedContext: - {realm} or {realmId}: the realm/database identifier - {tenantId}: the tenant identifier - {orgRefName}: the organization reference name - {accountId}: the account identifier - {ownerId}: the owner identifier

Example manifest usage:

Example dataset with variables:

After interpolation with tenantId="acme-corp" and realm="acme-realm":

Notes and guarantees: - Optional values missing from SeedContext are simply omitted; existing record values are preserved unless you target the same field - Transforms operate on in-memory maps and cannot perform I/O by default; keep them deterministic so re-runs are idempotent - Compose multiple transforms when needed (for example: first tenantSubstitution, then a custom id computation) - To add new transform types, implement SeedTransformFactory and register it during SeedLoader.builder() with registerTransformFactory("myType", new MyFactory()); then declare - type: myType in the manifest - To add custom variables for stringInterpolation, implement SeedVariableResolver and register it with the builder using addVariableResolver(resolver) or make it a CDI bean for automatic discovery

When to use transforms (and why): - Injecting tenant/realm identity: keep datasets source-controlled and generic; inject tenant IDs at apply time (tenantSubstitution) - Deterministic IDs: derive _id or other surrogate keys from naturalKey so upserts remain stable across environments - Normalization and defaults: add missing fields, convert formats, enforce enums before write - Reference remapping: translate human-readable codes in the dataset into datastore-specific identifiers (ObjectIds, UUIDs) in a repeatable way

Practical examples

1) Tenant identity injection (built-in) Manifest snippet:

Why: keep codeLists.ndjson portable across tenants; provisioning injects the right IDs based on SeedContext.

2) Deterministic _id from natural key (custom) - Goal: ensure stable MongoDB _id across re-applies and environments, derived from codeListName+code - Approach: implement a custom transform that computes a SHA-1/MD5 hash (or any deterministic function) and sets _id

Java registration:

Manifest usage:

Why: makes upserts resilient and allows cross-environment joins by a stable key.

3) Foreign key remapping by code (custom) - Goal: dataset uses human-readable statusCode; transform maps it to a canonical statusId - Approach: custom transform with an in-memory map or deterministic derivation

Manifest usage:

Why: keeps datasets human-friendly while persisting efficient identifiers.

4) Defaulting and sanitization (custom) - Goal: ensure missing fields get defaults and strings are trimmed/lowercased - Approach: simple custom transform that fills defaults and cleans values

Manifest usage:

Why: enforces consistency without editing large datasets.

Testing transforms: - Add integration tests that seed into an ephemeral DB and assert both record shape and _seed_registry entries - For custom transforms, add focused unit tests for edge cases (missing fields, nulls, unexpected types)

The SeedLoaderIntegrationTest demonstrates end-to-end seeding using the demo seed pack at src/main/resources/seed-packs/demo-seed.

What the test does: - Creates a SeedLoader backed by FileSeedSource, MongoSeedRepository, and MongoSeedRegistry. - Builds a SeedContext populated with tenant/realm details to exercise the tenantSubstitution transform. - Applies the pack reference SeedPackRef.of("demo-seed"), which resolves the latest version of that pack (1.0.0 in tests). - Asserts that 2 records were inserted into the codeLists collection and that the tenantSubstitution fields were populated from the context. - Verifies an entry was written to _seed_registry with the dataset checksum and records: 2. - Re-applies the same pack and asserts the record count remains 2, demonstrating idempotency (thanks to upsert + registry checksum).

Why these design choices: - NDJSON for datasets: allows streaming large datasets and simple line-by-line diffs in code review; arrays are also supported for smaller payloads. - Natural-key upsert: manifests declare naturalKey to form the filter for replaceOne(…​, upsert=true) ensuring idempotent writes and predictable overwrites. - Transform pipeline: keeps dataset files free of environment-specific values; all tenant/realm specifics are injected consistently at apply time. - Registry-based skip: checksums per dataset avoid unnecessary writes when content hasn’t changed—fast, safe re-runs during deployments. - Semantic-version selection: when multiple versions of a pack are available, the latest semver is used unless an exact version is requested.

Alternatives considered: - Store seed state in an external table keyed only by version. Rejected in favor of per-dataset checksums to detect content drift without bumping versions. - Hardcode seeding logic inside ad-hoc migrations. Rejected for lack of composability and poor reviewability. - Use inserts only (no upsert). Rejected due to lack of idempotence and difficulty correcting baseline data.

Archetypes are named compositions of seed packs that model a product edition, SKU, or vertical stack. Instead of listing several packs every time you provision a tenant, you define an archetype once in a manifest and then apply it by name.

What an archetype is in this context: - It lives inside a seed pack manifest under archetypes:. - It contains a list of includes (same syntax as top-level includes) referring to packs and version ranges. - When applied, the loader resolves those pack refs plus the hosting pack itself (the manifest that defines the archetype) so that local datasets are included as part of the archetype. - Resolution uses semantic version rules and deduplicates by pack name, respecting dependency order and preventing cycles.

When to use archetypes: - To represent product tiers (e.g., Community, Pro, Enterprise) that bundle different combinations of base packs and optional modules. - To group verticalized defaults (e.g., Logistics-Fulfillment, Healthcare-Core) without forcing consumers to know every underlying pack.

The framework exposes admin-only endpoints to inspect and apply seed packs per realm (tenant DB). These are disabled to non-admin users via role checks.

Base path: /admin/seeds

Endpoints: - GET /admin/seeds/pending/{realm} - Lists pending seed packs for the realm. A pack is pending if any dataset checksum differs from the last applied or was never applied. - Optional query parameter: filter=pack1,pack2 to restrict by pack name. - Response example:

+

Authentication and roles: - All endpoints require role admin. In integration tests you can use @TestSecurity(user="admin", roles={"admin").

Configuration: - quantum.seed.root: filesystem path to the root folder where seed packs are discovered. In tests this defaults to src/test/resources/seed-packs. - quantum.seed.apply.filter: optional comma-separated list of pack names to limit automatic application by changeset.

Note about collection vs modelClass: - You can now omit collection in a dataset when you specify modelClass. The framework will derive the collection name from the Morphia mapping of the model class. - Derivation rules: if the model class has @Entity with a non-empty value, that value is used as the collection name; otherwise the simple Java class name is used. - This derived name is used consistently for logging, in the _seed_registry entries, and for the Mongo fallback path. - Backwards compatibility: specifying collection is still supported, and will override the derived name.

In addition to the default Mongo-based persistence, you can instruct the seeding pipeline to persist a dataset via a Morphia repository mapped to a concrete UnversionedBaseModel. This enables: - Automatic class discriminator fields (for example, Morphia’s _t) and proper collection mapping. - Population of framework-managed fields (dataDomain, audit info, etc.) by the repository layer. - Consistent security filtering and validations applied by Morphia repos in normal runtime.

How to enable Morphia for a dataset: - Add modelClass to the dataset in manifest.yaml with the fully-qualified class name that extends UnversionedBaseModel. - Keep naturalKey and transforms as usual. The loader will still compute checksums and idempotently upsert.

Example manifest snippet:

Walkthrough (based on the integration test MorphiaSeedRepositoryIntegrationTest): - The test builds a SeedLoader with MorphiaSeedRepository and MongoSeedRegistry pointing at src/test/resources/seed-packs. - A SeedContext is created for a test realm. For repository-layer behavior (dataDomain, permissions), a minimal SecurityContext is also established in the test. - The loader applies SeedPackRef.of("morphia-demo"), which reads datasets/codeList.ndjson with two CodeList records. - The dataset declares modelClass, so MorphiaSeedRepository resolves the CodeList MorphiaRepo and attempts to save via Morphia. - If Morphia permission rules are not yet configured for the realm, MorphiaSeedRepository automatically falls back to a direct Mongo write, ensuring seeds still apply predictably but without Morphia-only fields. - The test then asserts that two documents exist in the CodeList collection and that _seed_registry has a matching history entry for the dataset checksum with records: 2.

Key behaviors and edge cases: - Indexes: When modelClass is present, ensureIndexes relies on Morphia mapping to enforce annotated indexes for the model. Any requiredIndexes declared in the manifest are still respected by the Mongo fallback. - Transforms: tenantSubstitution adds tenant context fields. When saving via Morphia, the repository adapts top-level tenant fields into dataDomain.* automatically for common models, and removes transient fields like realmId before mapping. - Permissions: If repository permissions prevent writes (for example, missing policies in a brand-new realm), MorphiaSeedRepository will log a warning and fall back to Mongo so seeding is not blocked. As you evolve policies, the Morphia path will be taken automatically on future runs. - Idempotency: Upsert semantics still honor naturalKey for both Morphia and Mongo paths, and _seed_registry records checksums so unchanged datasets are skipped.

See also: - Test source: quantum-framework/src/test/java/com/e2eq/framework/service/seed/MorphiaSeedRepositoryIntegrationTest.java - Manifest and dataset: quantum-framework/src/test/resources/seed-packs/morphia-demo - Admin APIs: /admin/seeds to list pending, apply packs, and inspect history per realm.

Starting with Quantum 1.2.2, dataset files in a manifest can be specified either as:

Notes: - If the dataset value contains "://", it is treated as a URL and routed by scheme. Otherwise it is treated as a relative path. - Using URLs allows you to mix sources inside a single manifest (e.g., some datasets from the local filesystem and others from S3). - This is extensible; additional schemes can be supported by adding optional modules in the future.

Examples:

Routing behavior: - Relative path: delegated to the SeedSource that loaded the manifest (unchanged behavior). - file:// URL: handled by FileSeedSource. - s3:// URL: handled by S3SeedSource (requires the optional quantum-seed-s3 module on the classpath).

The framework provides an optional SeedSource backed by Amazon S3, packaged in a separate module so you can include it only when needed: