Skip to content

Nuvom Architecture

This document explains the internal architecture of Nuvom, a lightweight, plugin-first task execution engine for Python.

High-Level Overview

Nuvom is designed to decouple task definition, discovery, execution, queuing, and result storage. Each layer is pluggable and follows a clearly defined contract via abstract base classes.

     +-------------------------+
     |      @task decorator    |
     +-------------------------+
                  |
                  v
      +------------------------+
      |     Task Registry      | <--- loaded from manifest
      +------------------------+
                  |
                  v
+-------------+     +-------------------+
| Dispatcher  | --> |  Job Queue        |
+-------------+     +-------------------+
                         |
                         v
            +----------------------+ 
            |   Worker Pool        |
            | (Threads + Runner)   |
            +----------------------+ 
                         |
                         v
            +----------------------+ 
            |  Result Backend      |
            +----------------------+

Core Components

@task Decorator

Location: nuvom/task.py

  • Wraps a function to register it as a Nuvom task.
  • Adds metadata (retries, timeout_secs, etc.).
  • Supports .delay() and .map() for job dispatch.
  • All tasks are auto-registered via AST and manifest system.

Task Discovery

Location: nuvom/discovery/

  • Uses AST parsing (not imports) to detect decorated @task functions.
  • Avoids side-effects, safe for large codebases.
  • Uses .nuvomignore to skip paths.
  • Output is cached in .nuvom/manifest.json for fast reloading.

Key files:

  • walker.py – file traversal
  • parser.py – AST parsing
  • manifest.py – manifest file I/O
  • auto_register.py – registry loader

Task Registry

Location: nuvom/registry/registry.py

  • Thread-safe global registry for tasks.
  • Validates task names (prevents duplicates unless force=True).
  • Used by the dispatcher and job runner to resolve function names.

Dispatcher

Location: nuvom/dispatcher.py

  • Orchestrates job submission: serializes, enqueues, retries.
  • Provides .delay(), .map(), and job creation utilities.
  • Uses msgpack for efficient, cross-platform job serialization.

Job Queues

Location: nuvom/queue_backends/

Built-in backends:

  • MemoryJobQueue
  • FileJobQueue
  • SQLiteJobQueue (v0.10)

Required interface methods:

enqueue(job)
dequeue(timeout=None)
pop_batch(batch_size)
qsize()
clear()

Custom backends can be added via the plugin system.

Workers & Job Execution

Location: nuvom/worker.py, nuvom/execution/job_runner.py

  • Each worker runs in its own thread.

  • Jobs are executed with timeouts, retries, and lifecycle hooks:

  • before_job()

  • after_job()
  • on_error()
  • ThreadPoolExecutor is used internally for concurrency.
  • Supports graceful shutdown with log flushing and safe teardown.

Result Backends

Location: nuvom/result_backends/

Built-in backends:

  • MemoryResultBackend
  • FileResultBackend
  • SQLiteResultBackend

All result backends implement:

set_result(job_id, ...)
get_result(job_id)
set_error(job_id, ...)
get_error(job_id)
get_full(job_id)
list_jobs()

Use .nuvom_plugins.toml to register custom plugins.

Logging

Location: nuvom/log.py

  • Unified logging across all modules using Rich.
  • Logs are styled, color-coded, and exception-aware.
  • Categories: debug, info, warning, error.

Plugin Architecture

Location: nuvom/plugins/

Nuvom supports plugins for:

  • Queues
  • Result backends
  • Monitoring/exporters

Plugins follow a strict Plugin protocol with start() and stop() lifecycle methods.

[plugins]
queue_backend = ["custom.module:MyQueue"]
result_backend = ["custom.module:MyResult"]

Each plugin must register itself via a Plugin subclass, and may use register_queue_backend() or register_result_backend().

Job Lifecycle

  1. Developer defines a task with @task.
  2. nuvom discover tasks parses and caches it.
  3. Job is queued with .delay() or .map().
  4. Worker dequeues the job.

  5. JobRunner:

  6. Triggers lifecycle hooks

  7. Executes task with timeout/retry logic
  8. Stores result or error
  9. Job metadata is saved in the selected result backend.
  10. Results are queried via SDK or CLI.

Design Principles

  • Plugin-first, interface-driven
  • No global daemons or dependencies like Redis
  • Developer-first: minimal config, rich logging, CLI tooling
  • Native on Windows, Linux, macOS
  • Built to teach: readable source, clean separation

For more, see: