Configuration

At startup, Magpie application will load multiple configuration files to define various behaviours or setup operations. These are defined though the following configuration settings presented below.

All generic Magpie configuration settings can be defined through either the magpie.ini file or environment variables. Values defined in magpie.ini are expected to follow the magpie.[variable_name] format, and corresponding MAGPIE_[VARIABLE_NAME] format is used for environment variables. Both of these alternatives match the constants defined in constants.py and can be used interchangeably. Order of resolution will prioritize setting values over environment variables in case of matching duplicates values.

Configuration Files

File: magpie.ini

This is the base configuration file that defines most of Magpie’s lower level configuration. A basic example is provided in magpie.ini which should allow any user to run the application locally. Furthermore, this file is used by default in each tagged Docker image. If you want to provide different configuration, the file should be overridden in the Docker image using a volume mount parameter, or by specifying an alternative path through the environment variable MAGPIE_INI_FILE_PATH.

File: providers.cfg

This configuration file allows automatically registering service definitions in Magpie at startup. When the application starts, it will look for corresponding services and add them to the database as required. It will also look for mismatches between the service name and URL with the corresponding entry in the database to update it to the desired URL. See MAGPIE_PROVIDERS_CONFIG_PATH below to setup alternate references to this type of configuration. Please refer to the heading of sample file providers.cfg for specific format and parameter details.

File: permissions.cfg

This configuration file allows automatically registering or cleaning permission definitions in Magpie at startup. Each specified permission update operation is applied for the corresponding user or group onto the specific service or resource. This file is processed after providers.cfg in order to allow permissions to be applied on freshly registered services. Furthermore, sub-resources are automatically created if they can be resolved with provided parameters of the corresponding permission entry. See MAGPIE_PERMISSIONS_CONFIG_PATH below to setup alternate references to this type of configuration. Please refer to the heading of sample file permissions.cfg for specific format details as well as specific behaviour of each parameter according to encountered use cases.

File: magpie.env

By default, Magpie will try to load a magpie.env file which can define further environment variable definitions used to setup the application (see MAGPIE_ENV_FILE setting further below). An example of expected format and common variables for this file is presented in magpie.env.example.

Important Notes:

If magpie.env cannot be found (using setting MAGPIE_ENV_FILE) but magpie.env.example is available (after resolving any previously set MAGPIE_ENV_DIR variable), this example file will be used to make a copy saved as magpie.env and will be used as the base .env file to load its contained environment variables. This behaviour is intended to reduce initial configuration and preparation of Magpie for a new user.

When loading variables from the .env file, any conflicting environment variable will NOT be overridden. Therefore, only missing but required values will be added to the environment to ensure proper setup of Magpie.

File: postgres.env

This file behaves exactly in the same manner as for magpie.env above, but for specific variables definition employed to setup the postgres database connection (see MAGPIE_POSTGRES_ENV_FILE setting below). File postgres.env.example and auto-resolution of missing postgres.env is identical to magpie.env case.

Settings and Constants

Environment variables can be used to define all following settings (unless mentioned otherwise with ‘constant’). These values will be used by Magpie on startup unless prior definition is found within magpie.ini.

Loading Settings

These settings can be used to specify where to find other settings through custom configuration files.

  • MAGPIE_MODULE_DIR (constant)
    Path to the top level Magpie module (ie: source code).
  • MAGPIE_ROOT (constant)
    Path to the containing directory of Magpie. This corresponds to the directory where the repository was cloned or where the package was installed.
  • MAGPIE_CONFIG_DIR
    Configuration directory where to look for providers.cfg and permissions.cfg files.
    (Default: ${MAGPIE_ROOT}/config)
  • MAGPIE_PROVIDERS_CONFIG_PATH
    Path where to find providers.cfg file. Can also be a directory path, where all contained .cfg files will be considered as providers files and will be loaded sequentially.
    Note:
    If a directory path is specified, the order of loaded configuration files is not guaranteed (depending on OS implementation).
    Please refer to providers.cfg for specific format details and loading methodology according to arguments.
    (Default: ${MAGPIE_CONFIG_DIR}/providers.cfg)
  • MAGPIE_PERMISSIONS_CONFIG_PATH
    Path where to find permissions.cfg file. Can also be a directory path, where all contained .cfg files will be considered as permissions files and will be loaded sequentially.
    Note:
    If a directory path is specified, the order of loaded configuration files is not guaranteed (depending on OS implementation). Therefore, cross-file references to services or resources should be avoided to ensure that, for example, any parent resource dependency won’t be missing because it was specified in a second file loaded after the first. Corresponding references can be duplicated across files and these conflicts will be correctly handled according to configuration loading methodology.
    Please refer to permissions.cfg for specific format details and loading methodology according to arguments.
    (default: ${MAGPIE_CONFIG_DIR}/permissions.cfg)
  • MAGPIE_INI_FILE_PATH
    Specifies where to find the initialization file to run Magpie application.
    Note:
    This variable ignores the setting/env-var resolution order since settings cannot be defined without firstly loading the file referenced by its value.
  • MAGPIE_ALEMBIC_INI_FILE_PATH
    Path to .ini file which defines an [alembic] section specifying details on how to execute database migration operations.
    (Default: ${MAGPIE_INI_FILE_PATH}) [section defined within magpie.ini]
  • MAGPIE_ENV_DIR
    Directory path where to look for .env files. This variable can be useful to load specific test environment configurations or to specify a local path while the actual Magpie code is located in a Python site-packages directory (.env files are not installed to avoid hard-to-resolve settings loaded from an install location).
    (Default: ${MAGPIE_ROOT}/env)
  • MAGPIE_ENV_FILE
    File path to magpie.env file with additional environment variables to configure the application.
    (Default: ${MAGPIE_ENV_DIR}/magpie.env)
  • MAGPIE_POSTGRES_ENV_FILE
    File path to postgres.env file with additional environment variables to configure the postgres connection.
    (Default: ${MAGPIE_ENV_DIR}/postgres.env)

Application Settings

Following settings are used to define values that are employed by Magpie after loading the Loading Settings.

  • MAGPIE_URL
    Full hostname URL to use so that Magpie can resolve his own running instance location.
    Note:
    If the value is not set, Magpie will attempt to retrieve this critical information through other variables such as MAGPIE_HOST, MAGPIE_PORT, MAGPIE_SCHEME and HOSTNAME. Modifying any of these variables partially is permitted but will force Magpie to attempt building the full URL as best as possible from the individual parts. The result of these parts (potential using corresponding defaults) will have the following format: "${MAGPIE_SCHEME}//:${MAGPIE_HOST}:${MAGPIE_PORT}".
    (Default: "http://localhost:2001")
  • MAGPIE_SCHEME
    Protocol scheme URL part of Magpie application to rebuild the full MAGPIE_URL.
    (Default: "http")
  • MAGPIE_HOST
    Domain host URL part of Magpie application to rebuild the full MAGPIE_URL.
    (Default: "localhost")
  • MAGPIE_PORT
    Port URL part of Magpie application to rebuild the full MAGPIE_URL.
    (Default: 2001)
  • MAGPIE_SECRET
    Port URL part of Magpie application to rebuild the full MAGPIE_URL.
    (Default: 2001)
  • MAGPIE_CRON_LOG
    Path that the cron operation should use for logging.
    (Default: "~/magpie-cron.log")
  • MAGPIE_DB_MIGRATION
    Run database migration on startup in order to bring it up to date using alembic.
    (Default: True)
  • MAGPIE_DB_MIGRATION_ATTEMPTS
    Number of attempts to re-run database migration on startup in cased it failed (eg: due to connection error).
    (Default: 5)
  • MAGPIE_LOG_LEVEL
    Logging level of operations. Magpie will first use the complete logging configuration found in magpie.ini in order to define logging formatters and handler referencing to the logger_magpie section. If this configuration fails, this variable is used instead to prepare a basic logger.
    (Default: INFO)
  • MAGPIE_LOG_LEVEL
    Specifies whether Magpie logging should also enforce printing the details to the console when using helpers. Otherwise, the configured logging methodology in magpie.ini is used (which can also define a console handler).
    (Default: False)
  • MAGPIE_LOG_REQUEST
    Specifies whether Magpie should log incoming request details.
    Note:
    This can make Magpie quite verbose if large quantity of requests are accomplished.
    (Default: True)
  • MAGPIE_LOG_EXCEPTION
    Specifies whether Magpie should log a raised exception during a process execution.
    (Default: True)
  • MAGPIE_UI_ENABLED
    Specifies whether Magpie graphical user interface should be available with the started instance. If disabled, all routes that normally refer to the UI will return 404, except the frontpage that will return a simple JSON description as it is normally the default entrypoint of the application.
    (Default: True)

Security Settings

  • MAGPIE_SECRET
    Secret value employed to encrypt user authentication tokens.
    Important Note:
    Changing this value at a later time will cause previously created user tokens to be invalidated. It is strongly recommended to change this value before proceeding to user accounts and permissions creation in your Magpie instance.
    (Default: "seekrit")
  • MAGPIE_COOKIE_NAME
    Identifier of the cookie that will be used for reading and writing in the requests from login and for user authentication operations.
    (Default: "auth_tkt")
  • MAGPIE_COOKIE_EXPIRE
    Lifetime duration of the cookies. Tokens become invalid after this duration is elapsed.
    (Default: None [infinite])
  • MAGPIE_ADMIN_USER
    Name of the default ‘administrator’ generated by the application.
    Note:
    This user is required for initial launch of the application to avoid being ‘looked out’ as routes for creating new users require administrative permissions and access rights. It should be used as a first login method to setup other accounts. It will also be used by other Magpie internal operations such as service synchronization and setup during the application startup. If this user is missing, it is automatically re-created on following start.
    (Default: "admin")
  • MAGPIE_ADMIN_PASSWORD
    Password of the default ‘administrator’ generated by the application.
    (Default: "qwerty")
  • MAGPIE_ADMIN_EMAIL
    Email of the default ‘administrator’ generated by the application.
    (Default: "${MAGPIE_ADMIN_USER}@mail.com")
  • MAGPIE_ADMIN_GROUP
    Group name of the default ‘administrator’ generated by the application.
    Note:
    To simplify configuration of future administrators of the application, all their inherited permissions are shared through this group instead of setting individual permissions on each user. It is recommended to keep defining such higher level permissions on this group to ease the management process of granted access to all their members.
    (Default: "administrators")
  • MAGPIE_ADMIN_PERMISSION
    Name of the permission used to represent highest administration privilege in the application.
    Except for some public routes, most API and UI paths will require the user to have this permission (either with direct permission or by inherited group permission) to be granted access to view and edit content. The group defined by MAGPIE_ADMIN_GROUP automatically gets granted this permission.
    (Default: "admin")
  • MAGPIE_ANONYMOUS_USER
    Name of the default user that represents a non logged-in user (ie: invalid or no authentication token provided).
    This user is used to manage “public” access to service and resources.
    (Default: "anonymous")
  • MAGPIE_ANONYMOUS_PASSWORD (constant)
    Password of the default unauthenticated user.
    This value is not modifiable directly and is available only for preparation of the default user on startup.
    (Default: ${MAGPIE_ANONYMOUS_USER})
  • MAGPIE_ANONYMOUS_EMAIL
    Email of the default unauthenticated user.
    (Default: "${MAGPIE_ANONYMOUS_USER}@mail.com")
  • MAGPIE_ANONYMOUS_GROUP (constant)
    This parameter is preserved for backward compatibility of migration scripts and external libraries.
    All users are automatically member of this group to inherit “public” permissions to services and resources.
    Important Note:
    To set “public” permissions, one should always set them on this group instead of directly on MAGPIE_ANONYMOUS_USER as setting them directly on this user will cause only him to be granted access to the targeted resource. In this situation, all other users would “lose” public permissions after they authenticate themselves in Magpie as they would not be recognized as MAGPIE_ANONYMOUS_USER anymore.
    (Default: ${MAGPIE_ANONYMOUS_USER})
  • MAGPIE_EDITOR_GROUP
    Unused for the moment.
    (Default: "editors")
  • MAGPIE_USERS_GROUP
    Name of the default group created to associate all users registered in the application.
    New users are created with this group.
    (Default: "users")
  • MAGPIE_USER_NAME_MAX_LENGTH
    Maximum length to consider as a valid user name. User name specified during creation will be forbidden if longer.
    Note:
    This value should not be greater then the token length used to identify a user to preserve some utility behaviour.
    (Default: 64)
  • MAGPIE_LOGGED_USER
    Keyword used to define route resolution using the currently logged in user. This value allows, for example, retrieving the user details of the logged user with GET /users/${MAGPIE_LOGGED_USER} instead of having to find explicitly the GET /users/<my-user-id> variant. User resolution is done using the authentication cookie found in the request. If no cookie can be found, it defaults to the MAGPIE_ANONYMOUS_USER value.
    Note:
    Because the user executing the request with this keyword is effectively the authenticated user, the behaviour of some specific paths can be slightly different than their literal user-id counterpart. For example, user details will be accessible to the logged user (he can view his own information) but this same user will receive an unauthorized response if using is ID in the path if he doesn’t have administrator privilege.
    (Default: "current")
  • MAGPIE_DEFAULT_PROVIDER
    Name of the provider used for local login. This represents the identifier that will be set to define who to differentiate between a local sign-in procedure and a dispatched one to one of the known External Providers.
    The default is the value of the internal package used to manage user permissions.
    (Default: "ziggurat")

Phoenix Settings

Following settings provide some integration support for Phoenix in order to synchronize its service definitions with Magpie services.

  • PHOENIX_USER
    Name of the user to use for authentication in Phoenix.
    (Default: "phoenix")
  • PHOENIX_PASSWORD
    Password of the user to use for authentication in Phoenix.
    (Default: "qwerty")
  • PHOENIX_HOST
    Hostname to use for Phoenix connection for authentication and service synchronization.
    (Default: ${HOSTNAME}")
  • PHOENIX_PORT
    Port to use for Phoenix connection for authentication and service synchronization.
    (Default: 8443)

PHOENIX_PUSH = asbool(os.getenv(“PHOENIX_PUSH”, True)) TWITCHER_PROTECTED_PATH = os.getenv(“TWITCHER_PROTECTED_PATH”, “/ows/proxy”) TWITCHER_PROTECTED_URL = os.getenv(“TWITCHER_PROTECTED_URL”, None)

Postgres Settings

MAGPIE_POSTGRES_USER = os.getenv(“MAGPIE_POSTGRES_USER”, “magpie”) MAGPIE_POSTGRES_PASSWORD = os.getenv(“MAGPIE_POSTGRES_PASSWORD”, “qwerty”) MAGPIE_POSTGRES_HOST = os.getenv(“MAGPIE_POSTGRES_HOST”, “postgres”) MAGPIE_POSTGRES_PORT = int(os.getenv(“MAGPIE_POSTGRES_PORT”, 5432)) MAGPIE_POSTGRES_DB = os.getenv(“MAGPIE_POSTGRES_DB”, “magpie”)

External Providers

In order to perform authentication in Magpie, multiple external providers are supported. By default, the ‘local’ provider is ziggurat which corresponds to the package used to manage users, groups, permissions, etc. internally. Supported external providers are presented in the table below, although more could be added later on.

Each as different configuration parameters as defined in MagpieSecurity and use various protocols amongst OpenID, ESGF-flavored OpenID and OAuth2. Further external providers can be defined using this module’s dictionary configuration style following parameter specification of Authomatic package used for managing this authentication procedure.

Category Provider
Open Identity (OpenID) OpenID
Earth System Grid Federation (ESGF) (1) German Climate Computing Centre (DKRZ)
French Research Institute for Environment Science (IPSL)
British Centre for Environmental Data Analysis (CEDA) (2)
US Lawrence Livermore National Laboratory (LLNL) (3)
Swedish Meteorological and Hydrological Institute (SMHI)
OAuth2 GitHub Authentication
WSO2 Open Source Identity Server
(1) extended variant of OpenID
(2) formerly identified as British Atmospheric Data Centre (BADC)
(3) formerly identified as Program for Climate Model Diagnosis & Intercomparison (PCMDI)
Note:
Please note that due to the constantly changing nature of multiple of these external providers (APIs and moved Websites), rarely used authentication bridges by the developers could break without prior notice. If this is the case and you use one of the broken connectors, summit a new issue.

GitHub Settings

To use GitHub authentication provider, variables GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET must be configured. These settings correspond to the values retrieved from following steps described in Creating an OAuth App.

Furthermore, the callback URL used for configuring the OAuth application on Github must match the running Magpie instance URL. For this reason, the values of MAGPIE_URL, MAGPIE_HOST and HOSTNAME must be considered.

WSO2 Settings

To use WSO2 authentication provider, following variables must be set:

  • WSO2_HOSTNAME
  • WSO2_CLIENT_ID
  • WSO2_CLIENT_SECRET
  • WSO2_CERTIFICATE_FILE
  • WSO2_SSL_VERIFY

To configure your Magpie instance as a trusted application for WSO2 (and therefore retrieve values of above parameters), please refer to WSO2 Identity Server Documentation.