goats_tom/

This is the main Django application powering GOATS. It extends TOMToolkit and includes all server-side logic, models, views, serializers, tasks, and static frontend assets.

Main Responsibilities

  • Extends TOMToolkit to provide Gemini-specific functionality.

  • Implements core data models for DRAGONS, GPP, and observation management.

  • Serves and manages all Django views, serializers, and forms.

  • Defines and registers DRF API endpoints.

  • Manages real-time communication using Django Channels and WebSockets.

  • Orchestrates background tasks via Dramatiq and Redis.

  • Handles file and recipe management for DRAGONS data reduction.

  • Stores and renders user-facing templates and custom template tags.

  • Interfaces with external services (GOA, GPP, TNS, Antares, Astro Data Lab).

  • Contains all frontend assets and JavaScript apps for dynamic UI behavior.

Built on TOMToolkit

GOATS is built on top of the TOMToolkit, a Django-based platform designed for developing Target and Observation Manager (TOM) systems in astronomy.

TOMToolkit provides core functionality such as:

  • Target management

  • Observation submission

  • Alert ingestion

  • Facility integration

GOATS extends this foundation with additional features and domain-specific logic for Gemini observatories, including custom APIs, reduction integration, and UI enhancements.

How GOATS Extends TOMToolkit

GOATS is implemented as a Django project that uses TOMToolkit apps and infrastructure. The base Django application lives in src/goats_tom/.

URL Routing

GOATS reuses and overrides TOMToolkit URL patterns. You can find the custom URL configuration in:

src/goats_tom/urls.py

We include TOMToolkit routes and extend or override views where necessary.

Views

We build on TOMToolkit’s class-based views and override behavior for custom UI or data needs. Most custom views live in:

src/goats_tom/views/

or in:

src/goats_tom/api_views/

Examples include:

  • Customized target and observation views

  • Additional views for DRAGONS reduction and file management

  • Viewsets for Django REST Framework APIs

Templates

Django uses APP_DIRS = True to load templates from app-specific directories. GOATS overrides TOMToolkit templates by mirroring their paths under:

src/goats_tom/templates/

For example, to customize the target detail page, GOATS overrides:

src/goats_tom/templates/tom_targets/target_detail.html

Custom templates follow TOMToolkit’s structure and are rendered automatically when placed in the correct location.

Template Tags

GOATS defines custom template tags in:

src/goats_tom/templatetags/

To use them in templates:

{% load goats_tags %}

These tags allow us to modify rendering logic without changing view logic.

Static Files

Custom JavaScript, CSS, and images are stored in:

src/goats_tom/static/

GOATS extends TOMToolkit’s default interface with custom UI behavior and styling.

Customizations Unique to GOATS

In addition to TOMToolkit functionality, GOATS adds several core components to support Gemini-specific workflows and data reduction pipelines.

Key Features

  • Integration with the Gemini Program Platform (GPP) via the gpp-client package.

  • DRAGONS data reduction task orchestration using Redis and Dramatiq.

  • File and recipe management system for data processing and calibration.

  • Real-time UI updates via WebSockets for notifications and file downloads.

  • Per-user credential storage to support authenticated interactions with external services.

JavaScript and async UX

  • GOATS uses JavaScript (vanilla + ES6) to dynamically update page content without requiring full reloads.

WebSockets and Channels

  • GOATS uses Django Channels to support real-time communication.

  • Redis is configured as the channel layer backend to broadcast updates between workers and browser clients.

  • Subscriptions are used for: - System-wide and per-user notifications - Long-running file processing updates - Background file download status

Dramatiq and Redis

  • All long-running or non-blocking work is delegated to Dramatiq background tasks.

  • Redis acts as the message broker for Dramatiq.

  • Common background tasks: - Launching DRAGONS reduction jobs - Importing external data - Downloading files - Processing observation metadata - Sending program notifications

DRAGONS Data Reduction

  • The DRAGONS interface is the largest JavaScript web application in GOATS.

  • It is implemented in src/goats_tom/static/js/dragons_app/ and dynamically manages all data reduction workflows.

  • When a user initiates a reduction session through the UI, a DRAGONS Run is created.

  • A run is a snapshot of the input files, parsed using Astrodata, which extracts metadata for filtering and grouping.

  • Files are automatically grouped and sorted based on their metadata, but users may also manually sort or filter them.

  • Users interact with the interface to: - Inspect and manage grouped files. - Select or override recipes. - Launch DRAGONS reductions on subsets of files.

  • The reduction pipeline includes: - File validation. - Recipe selection. - Job queuing via Dramatiq. - Live status updates through WebSockets.

  • Final results are stored and made available through the file manager interface.

Gemini Program Platform (GPP)

  • GOATS integrates with GPP via the gpp-client library.

  • Users can: - Authenticate with GPP. - View program and observation metadata. - Import program information directly into the TOM system. - Trigger ToOs.

Credential Manager

  • GOATS securely stores user credentials for external services.

  • Credentials are stored per-user in the database and scoped by service (e.g., GPP, TNS).

  • All credential input is handled via secure forms.

  • Tokens are used to authorize data pulls and API requests.

Tips for Working with TOMToolkit

  • Before adding new code, check if TOMToolkit already provides what you need.

  • When customizing views, prefer subclassing TOMToolkit views over rewriting them.

  • Use Django’s template inheritance and block overrides to customize UI.

  • Explore TOMToolkit’s source code and documentation for implementation patterns.

Resources

Files

  • api_views/: Django REST Framework API viewsets for all major data types and services, including DRAGONS, GPP, and external integrations.

  • antares_client/: Lightweight client for querying and consuming Antares alerts.

  • astro_data_lab/: Client code and configuration for interacting with Astro Data Lab.

  • astroquery/: GOATS-specific extensions for astroquery GOA.

  • brokers/: Alert broker integrations (currently Antares), including parsing and formatting of incoming alert data.

  • consumers/: Django Channels WebSocket consumers for real-time updates (e.g., DRAGONS progress, notifications, downloads).

  • facilities/: Facility-specific logic, including Gemini facility plugins and override hooks for TOMToolkit.

  • filters/: Custom filter backends and logic, especially for AstroData and ReducedDatum filtering using safe expression parsing.

  • forms/: Django forms for user authentication, external service logins, and GOA/GPP queries.

  • harvesters/: Polling-based external data harvester (e.g., TNS harvester).

  • logging_extensions/: Custom logging handlers and configurations for DRAGONS-specific event logging.

  • middleware/: Django middleware extensions for GOATS-specific workflows and background behaviors.

  • migrations/: Django migration files for GOATS models.

  • models/: Core data models for GOATS including DRAGONS runs, files, recipes, keys, and credential management.

  • ocs/: Parsers and helpers for communicating with the OCS.

  • processors/: Data processing pipelines, including spectroscopy support and reduction chaining

  • realtime/: Real-time state tracking logic for WebSocket-driven updates (e.g., download progress, notifications).

  • routing.py: Channel routing configuration for WebSocket consumers.

  • serializers/: DRF serializers for all public-facing data models and nested APIs.

  • static/: JavaScript, CSS, and image assets for GOATS frontend, including the DRAGONS reduction app and GPP interface.

  • tasks/: Dramatiq background task definitions (e.g., for DRAGONS reduction, file downloads).

  • templates/: All Django templates used or overridden in GOATS, organized by TOMToolkit app and component.

  • templatetags/: Custom Django template tags and filters (e.g., for rendering dynamic buttons, visualizers, and overrides).

  • tests/: Internal test tools for GOATS, organized by module. Includes factories/, test settings, and URL overrides. Does not contain the actual test files.

  • tns/: Client integration with the Transient Name Server (TNS) for report submission and validation.

  • utils/: Shared utility functions and helpers used across multiple GOATS modules.

  • views/: All Django views, including class-based and function-based views for targets, observations, data products, keys, and reduction workflows.

  • urls.py: Root URL configuration for the GOATS Django app.