Testing
Running Tests
# Run all tests (unit + integration)
npm test
# Run with watch mode
npm run test:watch
# Run persistence test
npm run test:persistence
# Run with JUnit XML output (same flags the GitLab pipeline uses)
npm test -- --reporter=verbose --reporter=junit --outputFile=junit.xml
# Run with coverage (what the CI test stage runs; fails below the floor in
# vitest.config.ts). Writes coverage/{cobertura-coverage.xml,coverage-summary.json}
npm run test:coverage
# Run Firecracker-specific tests (Linux only, requires setup)
NSYS_CORTEX_TEST_FIRECRACKER=1 npm test
# Run the Docker-mode integration test (needs a Docker daemon + `npm run build:image`)
NSYS_CORTEX_TEST_DOCKER=1 npm test
# Run the LLM-gated integration tests (real Anthropic API)
NSYS_CORTEX_INTEGRATION_LLM=1 \
NSYS_CORTEX_ALLOW_INSECURE_DEV=1 \
ANTHROPIC_API_KEY=sk-ant-... \
npx vitest run packages/cortex-runtime/src/__tests__/integration/
Coverage
npm run test:coverage runs the suite under the V8 coverage provider
(@vitest/coverage-v8) with all: true, so every source file under
packages/*/src counts whether or not a test touches it. The CI test stage runs
this form: the Lines : NN% figure feeds GitLab's coverage badge/column, the
Cobertura report annotates MR diffs line-by-line, and the thresholds in
vitest.config.ts fail the job if coverage drops below the floor (lines 73 %,
statements 73 %, branches 74 %, functions 72 % — set just under the latest
measurement; raise as gaps close, never lower to pass).
Latest measurement (2026-10-19, 554 tests): 76.5 % lines, 77.1 % branches, 77.0 % functions — first measurement (2026-08-30), before the chat + API-route suites, was 65.7 % / 76.5 % / 70.0 %:
| Package | Lines | Notes |
|---|---|---|
| cortex-tools | 88.1 % | every shell-facing tool, policies, validateToolInput, MCP defer ≥ 89 % |
| cortex-runtime | 85.7 % | agent loop 84 %, compaction 98 %, failover 97 %, session storage 94 %; providers 67–82 % (unmocked streaming paths) |
| cortex-shared | 84.4 % | config.ts 69 % (provider-config export paths) |
| cortex-host | 69.7 % | api/server.ts 85 % (was 50 %), jailer-config 100 %, vsock 87 %, dev sandbox 84 %; KVM-only files 19–47 % |
| cortex-cli | 48.2 % | chat.ts 89 % (was 0 %); exec.ts covered by the lifecycle-commands suite; shell.ts, index.ts, config.ts, install.ts still 0 % (see caveat) |
| cortex-agent | 31.1 % | executor.ts 49 %; index.ts, pty-manager.ts, health.ts 0 % (see caveat) |
Caveat — child processes are invisible to V8 coverage. The dev-sandbox E2E
tests spawn the compiled agent bundle, and the CLI smoke test runs dist/index.js
on plain node, so the agent's dispatch loop and the CLI entry are exercised
but not counted. The agent/CLI rows understate real coverage; they do not
overstate it — chat.ts, shell.ts and pty-manager.ts genuinely have no test
driving their logic.
Where the next tests should go, by risk: (1) provider streaming edge
cases (Anthropic tool-use/thinking round-trips, structured-output translation,
non-2xx paths); (2) pty-manager.ts / shell — broken twice in the
Firecracker rootfs without a test noticing; (3) the remaining CLI commands
(shell.ts, exec.ts, install.ts, config.ts, index.ts wiring) through
the same ChatDeps-style seam; (4) memory/search.ts, health.ts,
system-prompt.ts. (Done: chat.ts and the API routes — the Chat and
API-routes suites above.) The KVM-only host code is honestly testable
only on a KVM runner — that is the standing validation checklist, not a
unit-test gap.
Continuous Integration
The repo ships a GitLab pipeline at .gitlab-ci.yml running on every push and MR. Four stages on the node:24-bookworm image (Node 24 / Krypton — active LTS):
| Stage | Command |
|---|---|
install |
npm ci --prefer-offline --no-progress |
check |
npm run build (tsc -b per workspace; emits dist + checks types) |
build |
npm run build:agent (esbuild bundle; reuses dist from check) |
test |
npm test -- --reporter=verbose --reporter=junit --outputFile=junit.xml |
The JUnit XML is uploaded via artifacts.reports.junit so failures surface in the MR "Test Reports" tab. Firecracker + LLM integration tests skip cleanly without their gate env vars. See docs/development.md for the full pipeline structure.
Test Coverage
Shared (@nsys-cortex/shared)
| Suite | Tests | Description |
|---|---|---|
| Unit: Redact | 12 | Secret redaction patterns: Anthropic / OpenAI / xAI / Google / GitHub / AWS keys, PEM blocks, Bearer/Authorization headers, object property scrubbing, cycles |
| Unit: Env | 6 | buildAgentEnv() filter — credential vars dropped, allow-list preserved, NSYS_CORTEX_* secret-shaped variants dropped, overrides applied |
| Unit: Config edit | 15 | coerceValue (bool/null/number/JSON/string); getConfigValue dotted paths; setConfigValue (no mutation, intermediate creation + createdKeys typo signal, empty-segment reject); unsetConfigValue (remove + no-op); validateConfig (valid/invalid, catches bad set); redactConfig; read/write round-trip in an isolated NSYS_CORTEX_CONFIG_DIR asserting the file is 0o600 and that the real ~/.nsys-cortex/config.json is never the target |
| Unit: Protocol schemas | 4 | AgentCommandSchema keeps the file_read offset/limit window on parse (non-strict z.object strips unknown keys — a field missing from the Zod schema silently never reaches the executor); through the payload union a bad window can only arrive absent or as a non-negative integer; FileReadPayloadSchema constrains it directly; AgentResponseSchema keeps totalLines/truncated |
| Unit: Path-safety | 9 | safeRealpath() — plain files inside workdir, symlink escape, sibling-prefix, non-existent leaf/outside path, ../ escape, descriptive error; resolveAllowedPrefixesSync makes a workdir reached through a symlink (macOS /tmp, /var) accept in-jail paths and still reject escapes; an agent replacing its workdir with a symlink after the prefix was resolved cannot widen the jail (and safeRealpath never resolves prefixes itself) |
Agent (@nsys-cortex/agent)
| Suite | Tests | Description |
|---|---|---|
| Unit: Vsock client | 1 | Receive path against a real UNIX-socket server: a command whose multibyte character is cut across two socket reads is reassembled intact (StringDecoder) |
| Unit: Frame guard | 2 | frameResponse(): a response under the cap passes through unchanged; an oversized one (2 KiB of NUL JSON-escapes past a 4 KiB cap — the JSON-expansion class) is replaced by a RESPONSE_TOO_LARGE error the host's AgentResponseSchema parses, instead of a frame the host would drop |
| Unit: Executor read window | 10 | readLines() against real temp files: 1-based window + totalLines; clipped at EOF / past-EOF and limit: 0 → empty; final line without trailing newline and empty file counted correctly; splits on \n only (a lone \r is content); a 6 MiB single line is flagged truncated and the next line is still reached; lineCount distinguishes "no line fit" from "one blank line"; missing file → "File not found"; path outside the workdir refused; a symlink the agent later puts in place of its own workdir is not followed (jail widening); the executor refuses to construct on a workdir that is already a symlink (planted in an earlier session) — also with a trailing slash, which makes lstat follow the link — while a not-yet-existing workdir is fine |
Host (@nsys-cortex/host)
| Suite | Tests | Description |
|---|---|---|
| Unit: API client | 8 | Firecracker REST API client (mocked HTTP) |
| Unit: Vsock server | 16 | Message framing, timeouts, concurrent requests, socket file chmod 0o600, receive-buffer cap (MAX_MESSAGE_SIZE) drops an unbounded delimiter-less frame, assertUnixSocketPathFits (107-byte sun_path limit, byte- not char-counted), a multibyte character cut across two socket reads is reassembled (StringDecoder); a prober's second connection is closed and does not count as the agent leaving; a connection that never answered is superseded by the agent and the outstanding request is re-sent to it; an unanswered connection's close keeps the pending request, which the later-connecting agent answers; stop() cancels a pending waitForNeuron() at once; an unproven client overflowing the frame cap is dropped without failing the pending request; two overlapping waits are both cancelled by stop() |
| Unit: Install command | 8 | buildInstallCommand() — the one install line the CLI and the API share: npm in every mode; pip as-is in dev, --user --break-system-packages on the Alpine image (docker, firecracker); apt in dev, refused on the image naming the mode and the alternatives; shell metacharacters refused with the schema message; a scoped @scope/name quoted whole and a version specifier refused by design; an unknown manager refused (#31) |
| Unit: Validation | 10 | InstallPackageSchema rejects shell metacharacters / very long names; CreateNeuronSchema agentId regex; SessionMessageSchema.images (valid base64+mime accepted, data-URL prefix rejected, unsupported mime rejected, >4 images rejected) |
| Unit: Rate limiting | 7 | rateLimitConfig env parsing (defaults, overrides, 0 disables, non-numeric/negative fallback, 1s window floor); end-to-end boot: general budget exhaustion → 429 { code: RATE_LIMITED, requestId } + RateLimit headers; =0 disables the limiter |
| Unit: API routes | 27 | The route surface on an ephemeral port with a real dev neuron + a mock provider (createApiServer({ provider })): health/version/request-id echo; neurons create (validation 400, 201, list/get, 404), exec (200; unknown neuron → 500 envelope without a message), files write/read (relative paths resolve to the workspace, absolute inside it, missing ?path 400, missing file success:false), install metachar rejection, tools list/404, snapshot no-op, delete 204; sessions create/list/get/?history=N/delete/404s; /message runs the loop (response, usage, toolCalls, persisted history, session usage), 404s for unknown session / neuron, 400 for oversized schema or bad image; /run SSE ordering + per-event runId/sessionId; a failing provider → 502 RUN_ERROR on /message and a generic error frame on /run, never echoing the provider's message; a provider that fails after partial text → 502 with partialResponse, never a 200; a run stopped by the caller's maxTurns → 200 + stopReason: max_turns on /message and a max_turns SSE event (no error frame) on /run; an :agentId route param the create schema would reject (x%2F, %2e%2e, 65 chars, a space) → 400 VALIDATION_ERROR on GET/DELETE//tools, never a lookup; a second POST /neurons for a running id is 409 CONFLICT; a neuron whose agent was killed is 200 stopped without a pid, exec and /message on it are 409 NOT_RUNNING, and POST /neurons re-creates it |
| Unit: Orphan cleanup | 13 | Pending .neuron.json older than grace window → killed + removed; within grace → kept; cleanupOrphans() with pgrep stubbed to a throwaway sleep child the test owns: a pid vouched for only by a state file behind a planted symlink (naming a foreign id) is still killed — the sweep reads state through the same guarded reader as discovery; stale cleanup never unlinks the path a state file names (only the derived socket) on list() or get(); the stale-pending sweep leaves a sleep bystander pid alive (state file still removed) and SIGKILLs a child whose argv carries the agent-bundle marker and whose environment binds it to this overlay dir — a sibling's agent (marker, other dir) survives; a pid of -1/0/'123' — or a file that is not JSON — is never running but the dir is still listed as stopped; a dir named bad name or with a newline is not listed at all; get() hides a fresh pending file and sweeps a stale one (bystander untouched); an unparseable or year-ahead createdAt is stale, five seconds or thirty minutes ahead is not; a state file that is a link to a valid file, or is over 64 KiB, is ignored and the dir listed as stopped; a FIFO under the name is not a state file and listing returns at once |
| Unit: Dev sandbox create | 6 | create('foo') next to an existing Foo is refused with readdir stubbed to a case-insensitive store's answer, before any socket or agent is started — so Linux CI exercises the refusal too; an id with whitespace or a control character is refused before touching the filesystem (<store>/foo would be a whitespace-bounded token inside <store>/foo bar in ps -E output); a missing agent bundle leaves no listening server or socket file behind (no phantom running row); the bundle lookup comes right after the conflict check, so no overlay dir either; a spawn() that throws synchronously leaves no listening server or socket file and no overlay dir either; an asynchronous spawn failure (ENOENT) rolls back within the race, rejects with the spawn error itself, and leaves no overlay dir; a case-sensitive store keeps only Foo after the collision refusal; the bundle lookup is stubbed, so the file needs no built bundle; the agent is spawned with the host's own Node binary, with a PATH lookup only when that path is gone — a boot that then dies on exit 1 reports the exit, the error line and version trailer from a Node-shaped stderr (loader frames and a JSON log line's escaped frames dropped — the log line longer than the per-line cap, so its cut frames must not swallow the lines after it —, a 20 KB source line capped) and the fallback, and a boot without the fallback carries no such note |
| Unit: Firecracker process | 2 | FirecrackerProcess.spawn() without KVM (throwaway jailer under a temp chroot base, current uid): a missing jailer and one that exits at once both fail within milliseconds carrying the cause (ENOENT / exit code), and kill() removes the jail |
| Unit: Jail cleanup | 5 | Firecracker host-crash sweep against a fake cgroup tree: listCgroupLeafPids (pids per leaf, absent parent → empty), cleanStaleJailResources keeps the chroot whose cgroup leaf has pids and removes the rest / no-ops on a missing jail dir, isJailedVmmRunning probes via pgrep -f -- without throwing on regex metachars |
| Unit: Firecracker config | 20 | Pure jailer/isolation helpers (macOS-testable): chroot path derivation, cgroup args (memory.max incl. overhead, cpu.max per vCPU, pids.max), jailer argv (fc args after --, optional --netns, id sanitization), allocator index→uid/tap//30/MAC derivation, and the NAT command plan (tap owner uid, guest-/32 MASQUERADE, anti-spoof ! -s guest/32, metadata+CGNAT+RFC1918 egress DROPs, per-tap default-deny, IPv6 off + rp_filter, INPUT head-insert, inbound DROP, teardown mirrors with -D). Boot path itself requires KVM (not covered here). |
| Integration: Dev sandbox | 24 | Refuses a neuron whose overlay dir is a symlink (planted earlier) and discovery never follows such a link nor a state file naming a different id, nor touches the filesystem for a non-component id (x/ lstat-follows the link, .. is the store's parent; create() refuses both, and refuses foo next to an existing Foo on a case-insensitive store); the owning process leaves its own live neuron's socket and state file alone when that file is rewritten with a dead pid; create() over a .neuron.json that is a link to another file leaves that file untouched and ends with a regular state file; two concurrent create() calls for one id yield one neuron and one "already exists" rejection; an agent frozen, sent a request and then SIGKILLed (the only sequence that yields ECONNRESET on Linux) leaves the host alive, the in-flight request rejected, the neuron stopped and re-creatable; an agent that dies during the health check is rolled back with no socket or state file left; a pending file aged past the grace window while its own boot is in flight is neither judged nor swept by the owner; create/destroy, exec bash/node, file I/O, persistence, timeout, concurrency, workspace isolation, cross-process discovery; an agent SIGKILLed before answering during boot fails create() at once (the boot races the child's exit); a running id is a 409 conflict even when the agent bundle has gone missing |
| Integration: Firecracker | 4 | Boot time, exec, snapshot/restore refused with "not yet supported" under the jailer, crash recovery (skipped without NSYS_CORTEX_TEST_FIRECRACKER) |
| Integration: Docker | 6 | Real container: health + in-container workdir; runs as operator uid with CapEff 0, read-only image, private /tmp, only its own socket visible, no host env leak; path jail (/etc/passwd, traversal) + /home/agent/memory writable; workspace persists across destroy/create with host-uid ownership; discoverable + destroyable from a second manager (cross-process); an agent killed inside its container (docker kill) flips the neuron to stopped, execute() is a NotRunningError, the state file is released and create() replaces it. Skipped without NSYS_CORTEX_TEST_DOCKER=1 + the image |
| Unit: Docker argv/env | 20 | buildDockerRunArgs — trailing image node /opt/nsys-cortex/index.js; --cap-drop ALL + no-new-privileges + --read-only; operator uid (negative/fractional refused); memory = memory-swap, cpus, pids; exactly two --mounts (own home + own socket dir); tmpfs flags; name/labels/--rm/--init; env K=V with identifier check; readOnlyRoot=false only drops one flag; comma in mount src + whitespace in image refused. Network: bridge/none/custom ok, host/container: refused. Agent id ↔ container name round-trip, escape attempts refused, hostname ≤63. buildDockerAgentEnv forwards only locale/TZ/TERM, pins HOME/USER/SHELL, drops PATH/NODE_OPTIONS/secrets/host paths |
| Unit: Docker discovery | 22 | Through a fake docker CLI: cross-process running from state file + container; stale file dropped → stopped; pending orphan past grace reaped (rm -f) and reported stopped in the same list(); pending inside grace untouched + unlisted; overlay-only dir running iff labeled container; foreign/other-mode state files ignored; cross-process destroy() removes by name and keeps the overlay; destroy(id,false) wipes with nothing running; cleanupOrphans() removes only unknown labeled containers; same-named foreign container refused; invalid id rejected before any docker/fs call; CapacityError ceiling; network=host refused; socket parent 0700. Through a fake docker client (a shell wrapper the manager spawns; no daemon): a client that cannot start fails create() at once with the spawn error, a container that exits during boot fails it at once carrying docker's stderr (neither waits out bootTimeoutMs), a second create() for a boot in flight is a ConflictError while the owner neither lists nor sweeps its own pending file, and a stand-in agent that connects to the host socket, is frozen, sent a frame and SIGKILLed (ECONNRESET on Linux) leaves the host alive with both requests rejected, the neuron stopped with its state file and socket dir released, execute() a NotRunningError, and the next create() replacing it |
| Unit: Mode resolution | 10 | resolveRuntimeMode() with a probe seam: concrete modes pass through without probing and land in NSYS_CORTEX_MODE; auto → docker when the probe is ok, dev + fallbackReason otherwise; unset / empty env = auto; the env (where config.json's mode lands) is used when nothing is asked and an argument beats it; unknown mode refused naming auto, dev, docker, firecracker. probeDocker() against fake docker CLIs: missing binary → daemon not reachable (named, with the spawn error), daemon up but no image → image not found pointing at build:image, both present → ok echoing bin + image, defaults from NSYS_CORTEX_DOCKER_BIN/_IMAGE |
Toolkit (@nsys-cortex/tools)
| Suite | Tests | Description |
|---|---|---|
| Unit: Registry | 6 | Register/unregister, access policy filtering, dynamic policy update, bulk load, isToolAllowed |
| Unit: Policies | 21 | PathPolicy (3), CommandPolicy (5), ResourcePolicy (3), ToolAccessEngine (7 — incl. MCP allow-by-default, group:mcp deny, exact-name deny), CortexPolicyEngine (3) |
| Unit: Command policy hardening | 10 | rm -rf /usr blocked; bash -c sudo … recursive unwrap; base64-piped eval; curl | sh; $() and backtick substitution; safe commands not blocked |
| Unit: Core tools | 24 | Read (12 — incl. streamed offset/limit window via readFileLines, trailing newline not counted as a line in a whole-file read while blank lines at the end of a window are kept, executor size-cap refusal surfaced as isError, window past EOF reported instead of a fabricated blank line, executor-truncated window flagged with an unnumbered marker, a first line that alone exceeds the cap is an explicit error while a blank line followed by an oversized one renders normally (lineCount), Type.Integer window params coerce 2.5→2 / reject −1), Write (2), Edit (3), Bash (2), Grep (2), Find (2 — incl. -- before a flag-shaped glob), Ls (1) |
| Unit: Advanced tools | 19 | ApplyPatch (3), Process — split into 4 single-purpose tools (4), Memory_append/replace (4), Sessions (4), Elevated (3) |
| Unit: Apply-patch hardening | 4 | No-target-paths rejected; ../../etc/passwd rejected; absolute paths without git prefix rejected; clean in-workdir patch accepted |
| Unit: MCP adapter | 1 | Bogus stdio command surfaces a clean rejection |
| Unit: MCP env wiring | 8 | parseMcpServersEnv (valid stdio+http, invalid JSON/non-array → [], bad names skipped, missing transport fields skipped); loadConfiguredMcpTools (empty when unconfigured, broken server binary warns + continues); CortexConfigSchema.mcpServers (valid section accepted, bad name/url rejected) |
| Unit: MCP lazy loading | 10 | registerMcpTools threshold gating (eager ≤threshold, defer above, env 0=always/1000=never); mcp_find_tools meta-tool (inventory in description, * lists without activating, keyword activation is sticky + active thereafter, 8-per-search cap + overflow report, no-match message, idempotent re-search) |
| Unit: Session manager (persistence + expiry) | 13 | Persists on create/send/addHistory/destroy; rehydrates on initialize(); drops expired sessions during init; in-memory fallback when no storage; storage errors are non-fatal; idle/active state transitions; maxAgeMs makes session appear absent; maxHistoryEntries trims in-memory working set |
| Integration: Tools E2E | 11 | Write/read roundtrip, read pages with offset/limit through the real agent (window + totalLines survive the vsock schema), a NUL-heavy file under the byte cap errors instead of killing the neuron (frame guard; neuron still serves afterwards), edit, bash, grep, ls, memory persistence, policy blocks /etc, policy blocks dangerous commands, output truncation |
Runtime (@nsys-cortex/runtime)
| Suite | Tests | Description |
|---|---|---|
| Unit: Loop detector | 8 | Diverse calls, identical repeat, similar inputs, ping-pong, max calls, no-progress, reset, custom thresholds |
| Unit: Context manager | 7 | Token counting, message tokens, tool block overhead, compaction trigger, context state |
| Unit: Compaction | 8 | Preserve first+tail, custom preserveTail, tool_use/tool_result serialization, 50k truncation, summarization fallback, provider options, unchanged when small, empty middle |
| Unit: Tool-pair invariant | 10 | findToolUseIds / findToolResultIds; snapTailBoundary outward-snap; multi-pair sequences; assertToolPairsBalanced orphan/duplicate detection |
| Unit: Retry | 9 | isRetryable for 408/429/5xx/529 and Node net errors; withRetry success / retry-then-succeed / non-retryable / max-attempts / pre-aborted signal |
| Unit: Session storage | 13 | Save/load metadata, append/load history, list, delete, per-agent transcript namespacing, chmod 0o700 on storage dir, transcript + index files created 0o600, structured tool-call/tool-result/thinking field round-trip, isError serialized only when true, malformed JSONL line skip, rejects appendEntry without prior saveSession |
| Unit: Session reconstruction | 1 | End-to-end reload: a tool-using turn persisted by one manager+loop is reconstructed with its tool_use ↔ tool_result pair by a fresh manager+loop over the same FileSessionStorage; no balance/error events; newest user message last |
| Unit: SSE backpressure | 4 | writeSse() contract: immediate resolve on accepted write; pends until drain when the socket buffer is full (with listener cleanup); resolves on abort mid-drain; skips the write entirely when already aborted |
| Unit: Failover | 16 | FailoverProvider: primary serves; pre-first-yield hop; per-candidate model override; run-model inheritance; torn-output propagation (no hop after a yield); last-error on full failure; abort doesn't hop; describe(). parseFailoverSpec (3). Factory env-chain + explicit-provider bypass + unknown-provider fail-fast (3). Agent-loop retry wiring: 529 before first token retries and completes (1) |
| Unit: Structured outputs | 8 | OpenAI response_format body shape (name + strict defaults) and omission; Gemini responseJsonSchema + mime type, dropped when tools present; Anthropic forced-tool translation (synthetic tool + tool_choice, input re-emitted as text deltas, end_turn stop) and drop-guards for tools/thinking; agent-loop threading |
| Unit: Vision input | 5 | OpenAI user-image → image_url data-URL parts (text-only stays string); Gemini → inline_data parts; agent loop builds structured user message + persists attachments; reconstruction rebuilds the image message across a storage reload |
| Unit: Providers | 7 | Anthropic/OpenAI provider names, calibrated token counting (chars/3.5 for Anthropic), message tokens, tool block overhead |
| Unit: Provider conversion | 24 | String/block/mixed content counting (3.5-char rate), tool_result blocks, image blocks, empty content, tool_use overhead, factory creation (incl. vLLM/LM Studio/SGLang/Mistral name overrides), unknown provider, getTokenizer wiring (Anthropic → AnthropicTokenizer, OpenAI-compat family + Gemini → OpenAITokenizer, failover delegates to primary, ContextManager counts with the injected tokenizer) |
| Unit: Gemini provider | 8 | Provider name, token counting, string/block/tool_use/tool_result/image message counting, factory creation |
| Unit: OpenAI tokenizer | 5 | js-tiktoken exact counts on gpt-4o; empty string; unknown-model fallback to o200k_base; per-message overhead; gap-vs-heuristic on code |
| Unit: Agent loop | 4 | Single-turn text, multi-turn with tools, maxTurns limit, abort signal |
| Unit: Agent loop edges | 11 | Unknown tool, policy block, stream error event, provider throw, tool execution error, identical loop detection, token usage tracking, timeout, malformed JSON input, event ordering |
| Unit: Parallel groups | 2 | Smoke: parallelSafe/resourceKeys exposed through CortexToolRegistry |
| Unit: Extended thinking | 4 | thinking_delta events forwarded to caller; thinkingBudgetTokens propagated to provider; default temperature=0 when thinking off (1 when on); signatures preserved in assistant-history blocks across two-turn runs |
| Unit: Thinking cross-provider | 8 | OpenAI budget→reasoning_effort tier mapping; temperature dropped when thinking enabled (passed through otherwise); reasoning_content streamed as thinking_delta + thinking_end; reasoning_tokens surfaced. Gemini thinkingConfig.{thinkingBudget,includeThoughts} request shape; thinking off → no thinkingConfig; thought:true parts streamed as thinking_delta; thoughtsTokenCount surfaced. |
| Integration: Agent run (LLM-gated) | 1 | End-to-end agent run against real Anthropic. Asks the LLM to write a file; asserts the file lands in the overlay with the requested contents. Proves LLM → tool → vsock → executor → disk roundtrip. |
| Integration: SSE streaming (LLM-gated) | 1 | Boots the HTTP API on a random localhost port; consumes /sessions/:sessionId/run over fetch(). Asserts SSE event order (run_start → text_delta+ → run_complete) and that every event carries runId+sessionId. |
CLI (@nsys-cortex/cli)
| Suite | Tests | Description |
|---|---|---|
| E2E: CLI bundle smoke | 8 | Runs the compiled dist/index.js on plain node (the artifact bin/nsys-cortex executes — nothing else in the suite loads dist/). --version matches CORTEX_VERSION, --help shows usage + subcommands, dev-mode guard refuses without NSYS_CORTEX_ALLOW_INSECURE_DEV (pinned to NSYS_CORTEX_MODE=dev, with the "requested explicitly" hint). Mode resolution through a fake docker CLI: the default auto picks docker (list answers without the acknowledgement); auto without a daemon prints Docker sandbox unavailable (…) — falling back to dev mode and then refuses; NSYS_CORTEX_MODE=docker alone selects the docker manager (the --mode flag no longer defaults to dev); config get mode runs with no probe and no guard and shows the auto-created auto. Builds the bundle on demand if missing. |
| Lifecycle commands | 11 | execCommand, installCommand, destroyCommand and snapshotCommand against a fake sandbox manager with stdout/stderr spies. Destroy: an unknown id (no neuron, no overlay) is an error and destroy() is not called; --wipe calls destroy(id, false) and says the overlay was removed, otherwise where it is preserved. Snapshot: a NotSupportedError rejects the command and Snapshot created is never printed (#32). Install: a manager the mode cannot use (apt in docker) is refused before any neuron starts, exit 1; a non-zero package-manager exit shows the neuron's stderr and Failed to install … via pip: exit code 1, never unknown error; success reports on stderr and keeps the manager's stdout on stdout. Exec: a zero exit writes only the command's stdout to stdout and leaves process.exitCode unset; a non-zero exit becomes process.exitCode with the status line and the command's stderr on stderr; a transport failure is exit 1 with Error: …; a neuron another process started is re-created here once. The bundle smoke test proves the same through a real dev neuron (exit 7 → status 7, empty stdout) (#31). |
Chat (chat.ts) |
25 | chatCommand end to end without a TTY or LLM (ChatDeps seam; mock provider; real dev neuron for tools). Non-interactive: --message piped output, --json shape, stdin message, empty stdin error, --output-schema → responseFormat (+ unreadable file), --image attach (+ unsupported type), provider failure → stderr + exit 1, missing credentials → auth error with the setup hint / unknown provider → creation error, a write tool call landing in the overlay with toolCalls listed, a two-tool parallel turn keeping each call's own input/duration (ends matched by toolCallId). Resume: --continue replays history to the model, --session must exist and belong to the agent. Interactive (PassThrough readline in terminal mode): run + footer + 0o600 readline history, /help /model (list + switch reaches the provider) /cost /sessions /context /clear, /save (0o600 markdown; empty-session error), inline @image.png, Ctrl-C at the prompt exits 0, Ctrl-C mid-run aborts (signal reaches the provider) and returns to the prompt, resumed sessions replay the recent exchange; a neuron whose agent died mid-session is restarted before the next turn and the turn after the restart drives a tool call into the new agent; a failing restart shows the error and keeps the session alive |
| Security regression | 65 | One executable statement of the whole security posture (see Security Posture). Env scrubbing (buildAgentEnv — incl. dropping NSYS_CORTEX_API_TOKEN and token-bearing NSYS_CORTEX_MCP_SERVERS), secret redaction (+ end-to-end logger wiring), file modes (config / sessions / logs 0o600, dirs 0o700), workdir path jail (symlink + sibling-prefix escape, policy engine), central tool-input validation (validateToolInput coerces + rejects args against the TypeBox schema — the root-cause backstop; null on an optional field is absent, on a required field rejected), read/search/memory tool shell-escaping (escape path+glob+name via shq, coerce numeric fields, -e/-- option terminator against flag injection for both rg and fd; memory_get reads via validated readFile) and symlink-jail (read reads via validated readFile; grep/find/ls reject a root that resolves outside the jail), process pid/tail coercion, prototype-pollution guard on the config-path setter (__proto__/constructor/prototype rejected), command-policy bypass resistance (shell-wrapper / $(…) / backticks / <(…) / base64-eval / curl | sh), tool access + presets (elevated never default, readonly can't mutate, MCP carve-out both ways), end-to-end HTTP bearer-token auth (401/200 + envelope carries no message), the network bind guard (refuses non-loopback without a token), and the 500 error envelope (never echoes err.message/stack). Hermetic — no keys, no network beyond an ephemeral loopback port. |
The two LLM-gated tests skip unless NSYS_CORTEX_INTEGRATION_LLM=1 AND ANTHROPIC_API_KEY are set. To run them:
NSYS_CORTEX_INTEGRATION_LLM=1 \
NSYS_CORTEX_ALLOW_INSECURE_DEV=1 \
ANTHROPIC_API_KEY=sk-ant-... \
npx vitest run packages/cortex-runtime/src/__tests__/integration/
Totals
| Category | Pass | Skip |
|---|---|---|
| All | 554 | 6 (4 Firecracker-only + 2 LLM-gated integration) |