ARCHITECTURE_5_parameter_editor.md

Parameter Editor and Uploader Sub-Application Architecture

Overview

The Parameter Editor and Uploader sub-application is the core of the ArduPilot Methodic Configurator. It provides sequential configuration through intermediate parameter files, allowing users to view documentation, edit parameters, upload them to the flight controller, and save them to files. This systematic approach ensures methodical, traceable, and safe vehicle configuration.

Requirements

Functional Requirements

1. Sequential Configuration Management

   • Must present configuration steps in a logical, numbered sequence
   • Must advance through intermediate parameter files systematically
   • Must track configuration progress and completion status
   • Must allow skipping steps that don't apply to specific vehicles
   • Must support jumping to specific configuration steps when needed

2. Parameter File Operations

   • Must load and parse intermediate parameter files (.param format)
   • Must preserve comments and formatting in parameter files
   • Must validate parameter syntax and structure
   • Must support parameter file editing and modification
   • Must save changes back to parameter files with proper formatting

3. Parameter Table Management

   • Must display parameters in an editable table format
   • Must show current values, new values, units, and change reasons
   • Must support parameter addition and deletion
   • Must provide upload selection for individual parameters
   • Must highlight different parameter types (read-only, calibration, etc.)

4. Documentation Integration

   • Must display relevant documentation for each configuration step
   • Must provide links to blog posts, wiki pages, and external tools
   • Must show parameter-specific documentation and tooltips
   • Must support automatic documentation opening in simple mode
   • Must handle offline operation when documentation is cached

5. Flight Controller Integration

   • Must upload selected parameters to flight controller
   • Must verify successful parameter upload
   • Must handle flight controller resets when required
   • Must re-download and validate parameter values after upload
   • Must support retry mechanisms for failed uploads

6. Progress Tracking

   • Must display overall configuration progress
   • Must show completion status for each configuration phase
   • Must provide visual indicators for mandatory vs. optional steps
   • Must track which steps have been completed successfully
   • Must support configuration resumption from any point

7. Validation and Safety

   • Must validate parameter values against acceptable ranges
   • Must warn about potentially dangerous parameter combinations
   • Must prevent upload of invalid or out-of-range values
   • Must provide safety checks for critical parameters
   • Must support parameter rollback capabilities

8. Summary and Reporting

   • Must generate comprehensive configuration summary at completion
   • Must categorize parameters by type (default, calibration, etc.)
   • Must create summary files for different parameter categories
   • Must provide configuration backup and archival
   • Must support configuration comparison and analysis

Non-Functional Requirements

1. Performance

   • Parameter table should handle 1000+ parameters efficiently
   • Real-time validation should not cause UI lag
   • File operations should be fast and responsive
   • Documentation loading should not block the interface

2. Usability

   • Interface should be intuitive for users at all skill levels
   • Parameter editing should provide immediate feedback
   • Error messages should be clear and actionable
   • Progress indication should be accurate and informative

3. Reliability

   • Parameter uploads must be atomic and verifiable
   • Configuration state must be preserved across sessions
   • File operations must maintain data integrity
   • Recovery mechanisms must handle unexpected failures

4. Scalability

   • Must support different vehicle types and configurations
   • Must handle varying numbers of configuration steps
   • Must adapt to different parameter set sizes
   • Must support multiple simultaneous vehicle configurations

Architecture

Components

Main Parameter Editor

• File: frontend_tkinter_parameter_editor.py
• Purpose: Primary interface coordinating all parameter editor functionality
• Responsibilities:
  • Orchestrate the overall parameter editing workflow
  • Manage configuration step progression
  • Handle user interface layout and navigation
  • Coordinate between documentation, table, and progress components
  • Manage application state and configuration persistence

Collapsible Information Sections

• File: frontend_tkinter_collapsible_label_frame.py
• Purpose: Reusable ttk.LabelFrame with a persistent disclosure button and a content frame
• Responsibilities:
  • Hide and restore content with grid_remove() and grid() without recreating controls
  • Keep translated section titles visible and support keyboard toggling
  • Preserve per-section expansion state while progress and documentation update
  • Let the parameter editor's existing <Configure> bindings reclaim released vertical space
• Consumers: StageProgressBar and DocumentationFrame, both initially expanded
• Lifetime: Collapse choices last for the current editor window; they are not stored in program settings

Plugin Lifecycle During Step Changes

After saving pending edits, the parameter editor deactivates and destroys the outgoing plugin view before running the step-change workflow. Deactivation completes safety actions such as stopping motors; destruction removes controls, keyboard bindings, and polling timers before dialogs or transfers process Tk events. The existing layout remains available for reuse.

The resulting step gets a new plugin view even when its plugin configuration matches the outgoing one. Forced step refreshes follow the same lifecycle. If the workflow raises an exception, the editor restores the plugin for the current step before propagating the error. A failed safety stop prevents the workflow from starting. Diagnostics temporarily suspend plugin receivers while preserving the active view.

The editor delegates connection timers, retries and worker cancellation to ConnectionRecovery in frontend_tkinter_connection_recovery.py. UI callbacks display connection status and refresh FC cells while preserving local edits. FC operations retain explicit leases through nested Tk event loops; tools whose lifetime outlasts their opening callback release their lease when they close. Parameter upload previews and export windows use _retain_fc_operation_until_closed() to bind lease release to destruction of their root, rather than infer ownership from visibility. Unrelated report windows do not block recovery. An open bitmask editor blocks new connection checks and defers a successful worker's completion through RecoveryCallbacks.can_refresh until the dialog closes, keeping its row widgets alive even on macOS where the dialog does not take a grab. SuspendablePlugin.on_suspend(reason) distinguishes exclusive operations from connection recovery; motor tests stop before exclusive tools take reception and yield without commands during recovery. Both recovery eligibility and suspension use resolve_plugin_lifecycle() to select the preferred paired suspend/resume hooks or the legacy hide/show pair.

Calibration Navigation Lock

• File: frontend_tkinter_navigation_lock.py
• Purpose: Keep the active configuration context stable during calibration and configured file uploads
• Responsibilities:
  • Disable registered navigation, upload, file-browser, export, and auxiliary action controls
  • Preserve each control's existing disabled state and the step selector's readonly state
  • Restore controls only after every owner releases its lock, tolerating widget teardown
• Integration:
  • BaseWindow.navigation_lock supplies the shared frontend-only service
  • The parameter editor registers its header and footer controls and guards queued callbacks
  • Calibration views use plugins/frontend_tkinter_helpers.py to acquire before sending commands and release after completion, cancellation, failed startup, or teardown
  • Accelerometer and level calibration retain their locks through parameter readback and staging
  • RC calibration retains its lock while collecting and saving channel extremes
  • Compass calibration retains its lock until its progress popup closes
  • Configured file uploads retain their lock while a background transport worker runs; the modal Tk wait continues processing estimates and progress
  • Closing the parameter editor is deferred until a configured file upload finishes; other navigation locks leave application close available
  • Future asynchronous calibrations can use start_calibration_with_navigation_lock() and end_calibration_navigation_lock() without depending on ParameterEditorWindow
  • Calibration continue/cancel controls and application close remain available

Documentation Frame

• File: frontend_tkinter_parameter_editor_documentation_frame.py
• Purpose: Display and manage configuration step documentation
• Responsibilities:
  • Render documentation for current configuration step
  • Handle links to external documentation sources
  • Manage automatic documentation opening in simple mode
  • Display mandatory level and step-specific information
  • Handle documentation caching and offline operation

Parameter Table Editor

• File: frontend_tkinter_parameter_editor_table.py
• Purpose: Interactive table for parameter viewing and editing
• Responsibilities:
  • Display parameters in sortable, filterable table
  • Handle parameter value editing with validation
  • Manage parameter selection for upload
  • Provide parameter-specific tooltips and help
  • Handle parameter addition and deletion operations

Stage Progress Tracker

• File: frontend_tkinter_stage_progress.py
• Purpose: Visual progress tracking for configuration stages
• Responsibilities:
  • Display overall configuration progress
  • Show completion status for each phase
  • Indicate mandatory vs. optional steps
  • Provide visual feedback for current step
  • Handle progress state persistence

Configuration Steps Backend

• File: backend_filesystem_configuration_steps.py
• Purpose: Manage configuration step definitions and metadata
• Responsibilities:
  • Load configuration step definitions from JSON
  • Validate step sequences and dependencies
  • Handle step-specific documentation metadata
  • Manage step categorization and grouping
  • Support multiple vehicle type configurations

Parameter File Backend

• File: backend_filesystem.py
• Purpose: Handle parameter file operations and management
• Responsibilities:
  • Load and parse parameter files
  • Preserve comments and formatting
  • Validate parameter file syntax
  • Handle file backup and versioning
  • Support batch parameter operations

Data Flow

1. Initialization

   • Load vehicle directory and configuration
   • Initialize configuration steps for vehicle type
   • Load current parameter values from flight controller
   • Set up progress tracking and state management

2. Step Processing

   • Load current intermediate parameter file
   • Display relevant documentation
   • Present parameters in editable table
   • Allow user modifications and validation
   • Enable parameter selection for upload

3. Parameter Upload

   • Validate selected parameters
   • Upload parameters to flight controller
   • Handle required flight controller resets
   • Verify parameter upload success
   • Update progress and advance to next step

4. Completion

   • Generate comprehensive summary
   • Create categorized parameter files
   • Archive configuration for future reference
   • Provide completion report and next steps

Integration Points

• Flight Controller Communication: Provides parameter upload/download capabilities
• Component Editor: Receives vehicle-specific component configuration
• Directory Selection: Works within selected vehicle directory
• Documentation System: Integrates with online and cached documentation

Derived Parameter Awareness Workflow

run_initial_component_editor() applies project-creation preferences only for new projects. Existing projects open the Parameter Editor directly. The Parameter Editor returns an explicit action when the user requests component editing, and its caller reopens the editor at the same parameter step after the component window closes. The component window reports its close outcome; the application decides whether an initial setup close exits startup or an on-demand close returns to parameter editing. Each on-demand visit rebuilds the parameter model so evaluator references use the updated component data.

After the Component Editor closes and before the Parameter Editor opens, process_component_editor_results() recalculates forced/derived parameters and notifies the user which parameter files will be affected, so they know which steps to revisit.

Phase 1 - Pure computation (calculate_derived_and_forced_param_changes): For each file, deep-copies the loaded ParDict, computes forced/derived parameters, and compares the result against the unmodified in-memory original. Returns a dict[str, ParDict] of files that differ. self.file_parameters is never mutated.

Phase 2 - User notification (process_component_editor_results in __main__.py): If any files differ, a warning message listing the affected filenames is shown so the user knows which Parameter Editor steps to pay attention to. No data is applied or written at this point. The actual forced/derived values are re-computed step-by-step inside the Parameter Editor as each file is visited, because derived parameters can depend on FC parameter values that may change in every step, making it impossible to pre-compute them here. Empty dict -> no warning shown.

Separation from Project-Creation FC Import

The optional "use FC parameters" behavior during new project creation is intentionally separate from the derived-parameter confirmation workflow above.

• One-time project creation import (VehicleProjectCreator.create_new_vehicle_from_template): when use_fc_params=True, source FC values are substituted directly during the template copy step inside LocalFilesystem.copy_template_files_to_new_vehicle_dir(). Each .param file is parsed into a ParDict, both blank_change_reason and FC value substitution are applied in memory, and the result is written to the destination in a single pass — before re_init loads the directory.
• Regular editor flow (process_component_editor_results): only computes forced/derived changes and asks for confirmation before mutating in-memory data.

This separation keeps project initialization deterministic while preventing silent derived-parameter file writes in the normal editing flow.

Parameter Management System

Parameter Types

The system recognizes and handles different parameter categories:

• Default Parameters: Parameters at their default values (light blue background)
• Read-Only Parameters: System parameters that cannot be modified (red background)
• Calibration Parameters: Vehicle-specific sensor calibrations (yellow background)
• Configuration Parameters: User-configurable operational parameters
• Safety Parameters: Critical parameters affecting vehicle safety

Parameter Validation

• Range Checking: Validate parameters against min/max bounds
• Type Validation: Ensure correct data types (integer, float, etc.)
• Dependency Checking: Validate parameter interdependencies
• Safety Validation: Check for potentially dangerous combinations
• Format Validation: Ensure proper parameter naming and format

Upload Process

1. Pre-Upload Validation: Check all selected parameters
2. Batch Upload: Send parameters to flight controller efficiently
3. Reset Handling: Automatically reset FC when required parameters change
4. Post-Upload Verification: Download and verify all uploaded parameters
5. Retry Logic: Handle failed uploads with appropriate retry mechanisms

Documentation System

Documentation Types

• Blog Posts: ArduPilot forum methodic configuration posts
• Wiki Pages: Official ArduPilot documentation
• External Tools: Links to relevant external tools and calculators
• Parameter Documentation: Detailed parameter descriptions and usage

Documentation Management

• Automatic Opening: Simple mode opens all relevant documentation
• Contextual Help: Parameter-specific tooltips and help text
• Offline Support: Cached documentation for offline operation
• Version Compatibility: Documentation matched to firmware version

Progress Tracking System

Configuration Phases

The system organizes configuration into logical phases:

1. Initial Setup: Basic vehicle configuration
2. Hardware Configuration: Component-specific settings
3. Safety Configuration: Failsafe and safety systems
4. Tuning Configuration: Performance optimization
5. Advanced Configuration: Optional advanced features

Progress Visualization

• Phase Completion: Visual indicators for each phase
• Step Progress: Individual step completion tracking
• Mandatory Indicators: Clear marking of required vs. optional steps
• Current Position: Highlighting of current configuration step

Error Handling Strategy

• Parameter Errors: Detailed validation messages with correction guidance
• Upload Errors: Retry logic with exponential backoff and user notification
• File Errors: Graceful handling with backup and recovery options
• Documentation Errors: Fallback to cached or alternative sources

Testing Strategy

• Unit tests for parameter validation logic
• Integration tests for upload/download workflows
• UI tests for table editing and navigation
• File system tests for parameter file operations
• End-to-end tests for complete configuration workflows

File Structure

frontend_tkinter_parameter_editor.py                    # Main parameter editor
frontend_tkinter_parameter_editor_documentation_frame.py # Documentation display
frontend_tkinter_parameter_editor_table.py              # Parameter table editor
frontend_tkinter_stage_progress.py                      # Progress tracking
backend_filesystem_configuration_steps.py               # Step management
backend_filesystem.py                                   # Parameter file operations

Dependencies

• Python Standard Library:

  • tkinter for GUI components
  • json for configuration file handling
  • re for parameter parsing
  • webbrowser for opening documentation links

• Third-party Libraries:

  • tkinter.ttk for enhanced table components
  • tkinter.font for text formatting

• ArduPilot Methodic Configurator Modules:

  • backend_flightcontroller for FC communication
  • backend_filesystem for file operations
  • frontend_tkinter_base_window for GUI base classes
  • frontend_tkinter_rich_text for documentation display

Configuration File Management

Intermediate Parameter Files

Parameter files follow a specific naming convention:

• ##_descriptive_name.param (e.g., 01_initial_setup.param)
• Files are processed in numerical order
• Each file contains parameters for a specific configuration aspect
• Comments provide documentation and rationale for parameter choices

Summary Files

At completion, the system generates several summary files:

• complete.param: All parameters with change reasons
• non-default_read-only.param: Read-only parameters that differ from defaults
• non-default_writable_calibrations.param: Vehicle-specific calibrations that differ from defaults
• non-default_writable_ids.param: Vehicle-specific IDs that differ from defaults
• non-default_writable_non-calibrations_non-ids.param: Reusable configuration parameters that differ from defaults
• reusable.param: Reusable configuration parameters including reusable defaults

Configuration Steps Definition

Configuration steps are defined in JSON files:

• configuration_steps_ArduCopter.json
• configuration_steps_ArduPlane.json
• etc.

Each step definition includes:

• Step number and name
• Documentation links
• Mandatory level (0-100%)
• Associated parameter file
• Dependencies and prerequisites

Performance Optimization

Table Performance

• Virtual scrolling for large parameter sets
• Lazy loading of parameter documentation
• Efficient filtering and sorting algorithms
• Optimized redraw operations

File Operations

• Asynchronous file loading
• Incremental parsing for large files
• Efficient backup and versioning
• Batch operations where possible

Memory Management

• Efficient parameter storage
• Cleanup of unused resources
• Optimized GUI component lifecycle
• Memory-conscious documentation caching

Security Considerations

• Parameter validation to prevent dangerous values
• Safe handling of file operations
• Protection against malformed parameter files
• Secure documentation link handling
• Validation of flight controller responses