Services

This section describes Service and underlying Resource elements packaged with Magpie, as well as their respective functionalities and intended procedural behavior. Section Basic Service and Resource details first presents the general details to implement all Service implementations, followed by section Available Services which details specific implementations.

Basic Service and Resource details

Each Service is a specialization of a Resource (see: magpie.models.Service implementation). Therefore, they can be listed and searched for either using /services API routes (using service_name) or using /resources API routes (with their attributed resource_id), whichever is more convenient for the required needs.

On top of any Resource’s metadata, a Service provides specific information about its location, its remote synchronization method (if any), and its exposed endpoint. Another important detail about the Service is its type. This will not only dictate its purpose, but also define the whole schema of allowed Resource under it (if any), as well as every one of their Allowed Permissions.

The final distinction between a Service and generic Resource is their position in the hierarchy. Only Service-specialized Resource (literally resource_type = "service") are allowed to be placed at the top of the tree hierarchy. All generic Resource nodes must be nested under some root Service. Relative references are indicated using parent_id and root_service_id in corresponding Resource details.

Every Service type provided by Magpie must derive from magpie.services.ServiceInterface. Each specific implementation (see Available Services) serves to convert a given incoming HTTP request components (method, path, query parameters, body, etc.) into the appropriate Service, Resource and Permission elements. This ultimately provides the required elements to resolve ACL access of a Request User toward the targeted Resource according to its Effective Permissions.

In order to implement a new Service type, two (2) methods and a few attributes are required. The first method is magpie.services.ServiceInterface.permission_requested() which basically indicates how the HTTP request should be interpreted into a given Permission. The second is magpie.services.ServiceInterface.resource_requested() which similarly tells the interpretation method to convert the request into a magpie.models.Resource reference.

Whenever Effective Permission or ACL needs to be resolved in order to determine if a Request User can have access or not to a Resource, Magpie will employ the appropriate Service implementation and call the methods to process the result.

Changed in version 3.0: All the resolution of Access and Scope modifiers are automatically handled according to the applied Permission on the Resource hierarchy. Therefore, no specific action is required to support these features for new Service implementations. See Permission Definition and Modifiers for details.

On top of the above methods, the following attributes must be defined.

Attribute

Description

ServiceInterface.service_type
(str)

Defines the mapping of registered Service to the appropriate implementation type. Each implementation must have an unique value.

ServiceInterface.permissions
(List[Permission])

Defines the Allowed Permissions that can be applied onto the Service reference itself.

ServiceInterface.resource_types_permissions
(Dict[Resource, List[Permission]])

Map of the allowed children Resource under the Service and their corresponding Allowed Permissions for each case. Leaving this map empty will disallow the creation of any children Resource, making the Service the unique applicable element of the hierarchy. Note that each Resource implemented by a derived class of magpie.models.Resource also provides details about Allowed Permissions, their type and further nested children Resource.

child_structure_allowed
(Dict[Type[ServiceOrResourceType], List[Type[models.Resource]]])

Map of allowed Resource type nesting hierarchy for the Service. This controls whether some children Resource can be placed under another to limit creation only to cases that are relevant for the implemented Service.

configurable
(bool)

Parameter that controls whether the Service supports custom configuration, providing means to slightly alter, enable or disable certain behaviours of the parsing methodology of request into Resource and Permission components for the given Service.

See also

ServiceTHREDDS and ServiceGeoserver for examples of Service implementations that support custom configuration.

On top of the above definitions, any service that derives from magpie.services.ServiceOWS should also provide follow parameters for parsing OWS requests.

Attribute

Description

ServiceOWS.params_expected
(List[str])

Represents specific parameter names that can be preprocessed during HTTP request parsing to ease following resolution of ACL use cases.

ServiceOWS.service_base
(str)

Provide the reference OWS type for handling requests with proper parsers and resolvers.

Furthermore, some Services specifically implement extended OWS utilities offered by GeoServer. They derive from ServiceGeoserverBase and should provide the following additional parameters.

Attribute

Description

ServiceGeoserverBase.resource_scoped
(bool)

Indicates if the Service is allowed to employ scoped models.Workspace naming, meaning that a Resource of that type can be extracted either from the request path or the specific request parameter using notation format <WORKSPACE>:<RESOURCE_PARAM>.

ServiceGeoserverBase.resource_multi
(bool)

Indicates if the Service supports multiple simultaneous Resource references within a single request (see Permissions Resolution for more details`), which must all be considered for Effective Resolution.

ServiceGeoserverBase.resource_param
(Union[Str, List[Str]])

Name of one or many request query parameter(s) from which to extract multiple equivalent Resource references when the Service supports multiple representations or notation conventions.

Changed in version 3.21: - Attribute ServiceOWS.params_expected has been moved from ServiceInterface to instead be directly under ServiceOWS since it applies only to derived classes from that base.

Added in version 3.21: - Attribute ServiceInterface.child_structure_allowed - Attribute ServiceInterface.configurable - Attribute ServiceOWS.service_base - Attribute ServiceGeoserverBase.resource_scoped - Attribute ServiceGeoserverBase.resource_multi - Attribute ServiceGeoserverBase.resource_param

Available Services

See also

Module magpie.services contains the implementation of every Service presented below.

See also

Multiple of the following implementation details require comprehension of the concepts introduced in Permission Definition and Modifiers. Consider looking this up if descriptions feel hard to understand at first glance.

ServiceAccess

The implementation of this Service is handled by class magpie.services.ServiceAccess. It is intended to act as a simple all-or-nothing endpoint barrier, where only Permission.ACCESS can be applied, and only directly on the Service itself. A User or Group that has that Permission with Access.ALLOW will be able to reach the Service. Any other operation (or explicit Access.DENY) will all result into denied access to the private URL registered by the Service.

Changed in version 3.0: This Service implementation dates prior to the integration of Access and Scope concepts that now provides modifiers to the Permission resolution methodology, notably prior to the addition of Access.DENY. This Service is equivalent to a simplified version of the fine-grained ServiceAPI alternative where no children Resource would be registered.

Note

Due to the single-level aspect of this Service, the Scope Permission modifier does not have any effect.

ServiceAPI

The implementation of this Service is handled by class magpie.services.ServiceAPI. It refers to a remote URL endpoint that should have a Resource tree formed out of the path segments. The Service only has one (1) type of Resource, namely magpie.models.Route, that can have an unlimited amount of nested children of the same type. The Allowed Permissions for this Service are Permission.READ and Permission.WRITE. All requests using GET or HEAD are mapped to Permission.READ access while all other HTTP methods represent a Permission.WRITE access.

Request path segments follow the natural hierarchy of the nested magpie.models.Route under the Service. For example, a proxy employing magpie.adapter.MagpieAdapter such that {PROXY_URL} is the base of the request path and the magpie.services.ServiceAPI named SomeAPI was registered would be mapped against each sub-Resource as presented below.

{PROXY_URL}/SomeAPI/some-resource/child_resource/final
{PROXY_URL}/SomeAPI/other-resource/subResource

SomeAPI                     [service: ServiceAPI]
    some-resource           [resource: Route]
        child_resource      [resource: Route]
            final           [resource: Route]
    other-resource          [resource: Route]
        subResource         [resource: Route]

Every magpie.models.Route as well as the Service itself can have the Permission based on the HTTP method of the incoming request. All Access and Scope modifiers are also supported for highly customizable ACL combinations. See Permission Definition and Modifiers for further details.

ServiceTHREDDS

The implementation of this Service is handled by class magpie.services.ServiceTHREDDS. It refers to a remote data server named Thematic Real-time Environmental Distributed Data Services (THREDDS). The Service employs two (2) types of Resource, namely magpie.models.Directory and magpie.models.File. All the directory resources can be nested any number of times, and files can only reside as leaves of the hierarchy, similarly to a traditional file system. The Allowed Permissions on both the Service itself or any of its children Resource are Permission.BROWSE, Permission.READ, and Permission.WRITE (see note below regarding this last permission).

Added in version 3.1: The Permission.BROWSE permission is used to provide listing access of contents when targeting a Resource of type magpie.models.Directory. When targeting a magpie.models.File, it instead provides metadata access to that file.

Permission Permission.READ can be applied to all of the resources, but will only effectively make sense when attempting access of a specific Resource of type magpie.models.File.

Changed in version 3.1: Permission Permission.READ does not offer metadata content listing of magpie.models.Directory anymore. For this, Permission.BROWSE should be used instead. Setting Permission.READ on a directory will only be logical when combined with Scope.RECURSIVE, in which case Magpie will interpret the Effective Permission to allow read access to all magpie.models.File under that directory, at any depth level, unless denied by a lower-level specification.

Finally, Permission.WRITE can also be applied on all of the resources, but are not explicitly employed during parsing of incoming requests.

Note

The Permission.WRITE is not handled by ServiceTHREDDS itself during ACL resolution as it is not considered by magpie.services.ServiceTHREDDS.permission_requested() that never returns this value. The method only returns either Permission.BROWSE or Permission.READ. A User or Group can still have this Applied Permission to allow a third party service to interrogate Magpie API about the presence of Permission.WRITE permission and perform the appropriate action with the result. The Effective Permission API routes will provide the resolved access. It is only Twitcher proxy that will not be able to make use of it during incoming requests as it depends on magpie.adapter.magpieowssecurity.MagpieOWSSecurity, which in turn employs the result from the ACL.

The Permission.WRITE is mostly preserved for backward compatibility of services that employed ServiceTHREDDS to obtain information about which directory or files (which registered Magpie Resource) are writable or not, although using another upload methodology that is not specifically executed via the actual remote THREDDS service.

As presented above, the main two permissions are Permission.BROWSE and Permission.READ which correspondingly serve to retrieve metadata and actual data of a given Resource. To distinguish requests between these two types of contents, ServiceTHREDDS employs two parts from the request path, the sub-path prefix and the file extension. The default methodology employed to categorize these two types of content is represented by the below configuration.

Note

A custom categorization between metadata and data contents can be provided With either the providers.cfg or a Combined Configuration File as described in greater lengths within the Configuration chapter.

providers:
  LocalThredds:
    # minimal configuration requirements (where the real `THREDDS` service resides)
    # other optional parameters from `providers.cfg` can also be provided
    url: http://localhost:1234
    type: thredds

    # customizable request parsing methodology (specific to `thredds` type)
    configuration:
      # path prefix to skip (strip) before processing the rest of the path in case the
      # registered service URL in Magpie does not have the same root as proxied by Twitcher public URL
      skip_prefix: thredds
      # define which pattern matches that will map different path variations into same file resources
      # this can be used to consider two file extensions as the same resource to avoid duplication of permissions
      file_patterns:
        # note: make sure to employ quotes and double escapes to avoid parsing YAML error
        #       patterns are **NOT** UNIX filters, but regex format (eg: dot is 'any-character', not a literal dot)
        - ".*\\.nc"
      # path prefix to resources to be considered as BROWSE-able metadata (directory listing or file details)
      metadata_type:
        prefixes:
          - null  # note: special YAML value evaluated as `no-prefix`, use quotes if literal value is needed
          - "catalog\\.\\w+"  # note: special case for `THREDDS` top-level directory (root) accessed for `BROWSE`
          - catalog
          - ncml
          - uddc
          - iso
      # path prefix to resources to be considered as READ-able data (i.e.: file contents)
      data_type:
        prefixes:
          - fileServer
          - dodsC
          - dap4
          - wcs
          - wms

Warning

Regular Expression Patterns

Ensure to properly escape special characters, notably the dot (.), to avoid granting unexpected permissions that would match any character. Format employed in above patterns are traditional Regex, not UNIX style filters.

Changed in version 3.2: Added catalog specific patterns by default to metadata prefixes that composes another valid URL variant to request Permission.BROWSE directly on the top-level THREDDS service (directory), although <prefix_type> is otherwise always expected at that second position path segment after the service name (see below example). The pattern allows multiple extensions to support the various representation modes of the catalog listing (e.g.: XML, HTML, etc.).

As of that version, the prefixes entries also support patterns, using standard regular expression syntax.

Changed in version 3.3: Added skip_prefix to allow ignoring intermediate path segments between the service name and the desired <prefix_type> position. A typical use case with THREDDS is the /thredds prefix it adds between its API entrypoint and Tomcat service running it. If this feature is not needed, it can be disabled by setting the parameter to null.

Assuming a proxy intended to receive incoming requests configured with magpie.adapter.MagpieAdapter such that {PROXY_URL} is the base path, the following path would point toward the registered service with the above YAML configuration.

{PROXY_URL}/LocalThredds

An incoming request will be parsed according to configured values against the following format.

{PROXY_URL}/LocalThredds[/skip/prefix]/<prefix_type>/.../<file>

The above template demonstrates that Magpie will attempt to match the <prefix_type> part of the request path with any of the listed prefixes in the configuration (metadata or data). The <prefix_type> location in the path (i.e.: which segment to consider as <prefix_type>) will be determined by the next part following configuration value defined by skip_prefix (any number of sub-parts). If skip_prefix cannot be located in the request path or was defined as null, the first part after the service name is simply assumed as the <prefix_type> to lookup.

If a match is found between the various prefixes and <prefix_type>, the corresponding metadata or data content will be assumed, according to where the match entry was located, to determine whether the requested Resource should be validated respectively for Permission.BROWSE or Permission.READ access. If no <prefix_type> can be resolved, the Permission will be immediately assumed as Access.DENY regardless of type. To allow top-level access directly on the Service’s root without <prefix_type>, it is important to provide null within the desired prefixes list. Duplicates between the two lists of prefixes will favor entries in metadata_type over data_type.

After resolution of the content type from <prefix_type>, the resolution of any amount of magpie.models.Directory Resource will be attempted. Any missing children directory Resource will terminate the lookup process immediately, and ACL will be resolved considering Scope.RECURSIVE of any applicable parent Resource for the given Permission selected by <prefix_type> and from where lookup stopped.

Once the last element of the path is reached, the file_patterns will be applied against <file> in order to attempt extracting the targeted magpie.models.File Resource. Patterns are applied until the first positive match is found. Therefore, order is important if providing multiple patterns. For example, if the path ended with file.ncml and, that both .*\\.ncml and .*\\.nc where defined in the configuration in that specific order, the result will first match .*\\.ncml, and the final magpie.models.File Resource value will be considered as file.ncml for lookup. In this case, another request using only file.nc would lookup an entirely different Resource, since the second pattern would be the first successful match. On the other hand, if only .*\\.nc was defined in file_patterns, the matched pattern would convert both the names file.ncml and file.nc to file.nc, which will lookup exactly the same magpie.models.File reference. The most common usage of this feature is to support additional extension suffixes as the same file with regrouped permissions such that file.nc, file.nc.ascii?, file.nc.html, etc. all correspond to single and common Resource.

To summarize, if file_patterns produces a match, that matched portion will be used as lookup value of the Resource. Otherwise, if not any match could be found amongst all the file_patterns, the plain <file> name is used directly (as is from the specified request path). The plain name is also used if file_patterns is explicitly specified as an empty list or null. Not explicitly overriding the field will result into using the above default file_patterns. The file_patterns allow for example to consider file.nc, file.ncml and file.nc.html as the same Resource internally, which avoids duplicating Applied Permission across multiple Resource for their corresponding metadata or data representations.

ServiceWFS

See also

This implementation is defined by magpie.services.ServiceWFS. It implements the original standard WFS definition. There is NO concept of magpie.models.Workspace for this Service implementation. Features are accessed directly using magpie.models.Layer typed Resources.

ServiceGeoserverWFS

See also

This implementation is defined by magpie.services.ServiceGeoserverWFS. It implements some extensions to base WFS by providing more Permission and scoping of magpie.models.Layer under magpie.models.Workspace in a file-system-like fashion.

ServiceBaseWMS

See also

Derived implementations: - ServiceGeoserverWMS - ServiceNCWMS2

This is a partial base class employed to represent OWS Web Map Service extended via other complete classes. It cannot be employed directly as Service instance. The derived classes provide different parsing methodologies and children Resource representation according to their respective functionalities.

It provides support for the following permissions, each corresponding to the appropriate functionality of WMS:

Similar to any other OWS based Service, the HTTP request takes a request query parameter that indicates which of the above Permission is being requested.

ServiceGeoserverWMS

See also

This implementation is defined by magpie.services.ServiceGeoserverWMS. It extends the base class by using children Resource defined by magpie.models.Workspace, which supports the same set of Permission as their parent Service. Each of those magpie.models.Workspace correspond to the equivalent element provided to GeoServer based HTTP request using query parameter layers, following format layers=<Workspace>:<LayerName>. The Permission is obtained from the request query parameter.

Warning

As of latest version of Magpie, there is no specific handling of the specific LayerName part of the targeted Resource. Please submit an issue with specific use-case if this is something that would be required.

ServiceNCWMS2

See also

This implementation is defined by magpie.services.ServiceNCWMS2. It extends the base class by using children Resource defined as magpie.models.Directory and magpie.models.File instances but, using the corresponding Permission entries from ServiceBaseWMS class instead of the default Permission.READ and Permission.WRITE (i.e.: see ServiceTHREDDS). The general idea is that the remote ncWMS2 service provider being represented by this Service points to the same NetCDF file resources as offered by THREDDS, but for mapping display. The HTTP request therefore points toward another proxy endpoint and employs different query parameters specific to WMS requests (instead of THREDDS), although the provided file reference is technically the same. For this reason, the same Resource hierarchy is supported, with any number of nested magpie.models.Directory and magpie.models.File as leaves. The targeted Resource by the HTTP request is extracted from either the dataset, layername or layers query parameter formatted as relative file path from the THREDDS` root. The applicable query parameter depends on the appropriate :term:`Permission` being requested based on the provided ``request query parameter.

Note

Although the class name employs NCWMS2, the registered type is represented by the string ncwms for executing requests toward the Magpie API and contents returned in its responses.

ServiceWPS

See also

Consider using ServiceGeoserver for multi-OWS implementation support under a common endpoint representing a GeoServer instance.

The implementation of this Service is handled by class magpie.services.ServiceWPS. It is intended to control access to the operations provided by an OWS Web Processing Service. This Service allows one (1) type of child Resource, namely the magpie.models.Process which represent the execution units that are registered under a remote WPS. Every magpie.models.Process cannot itself have a child Resource, making ServiceWPS maximally a 2-tier level hierarchy.

There are three (3) types of Allowed Permission which each represent an operation that can be requested from it (via request query parameter value of the HTTP request), specifically the Permission.GET_CAPABILITIES, Permission.DESCRIBE_PROCESS, and Permission.EXECUTE. The Permission.GET_CAPABILITIES corresponds to the retrieval of available list of Processes on the WPS instance, and therefore, can only be applied on the top-level Service. The other two permissions can be applied on either the Service or specifically on individual magpie.models.Process Resource definitions. When applied to the Service with Scope.RECURSIVE modifier, the corresponding Permission becomes effective to all underlying Processes. Otherwise, the Permission applied on specific magpie.models.Process entries control the specific ACL only for it. When a specific Permission is involved on a magpie.models.Process during Effective Permission resolution, the value of the query parameter identifier is employ to attempt mapping it against an existing Resource. The resolution of Effective Permissions in the event of multi-level tree Resource is computed in the usual manner described in the Permissions chapter.

Warning

When applying permissions on a per-magpie.models.Process basis, Scope.MATCH modifier is recommended if only a specific request should be granted access, although Scope.RECURSIVE would have the same effect currently because these resources are necessarily leaf nodes by definition. This is to prevent unexpectedly granting additional lower-level Resource access in the event this definition gets modified or extended in the future.

ServiceGeoserverWPS

See also

Consider using ServiceGeoserver for multi-OWS implementation support under a common endpoint representing a GeoServer instance.

The implementation of this Service is handled by class magpie.services.ServiceGeoserverWPS. It offers similar operations and request handling to ServiceWPS, but adds scoped magpie.models.Workspace definition of magpie.models.Process as for other GeoServer base Services, in order to properly parse request paths that include such references. Other than the Resource hierarchy being nested by magpie.models.Workspace, the rest of the parsing and handling methodology is equivalent to ServiceWPS.

ServiceGeoserver

Added in version 3.21.

See also

This Service is combined OWS implementation for GeoServer that allows simultaneous representation of WFS, WMS and WPS Resources all nested under a single reference hosted under common remote URL. Using this implementation, magpie.models.Workspace are first required as immediate children under the root Service, and can be followed by both Resources of type magpie.models.Layer and magpie.models.Process. There are two main advantages of using this combined implementation over their specific OWS counterparts.

First, using the same Resources to represent corresponding elements in the GeoServer across OWS endpoints reduces the chances of inconsistent access to otherwise equivalent Resources. For example, a magpie.models.Layer that is granted Permission to retrieve features (WFS) or to render their map raster (WMS) will be managed using the same reference. Using distinct ServiceGeoserverWFS and ServiceGeoserverWMS Service in Magpie would require from to administrator to always maintain their Permissions in sync. This is prone to many errors and confusion when managing multiple large hierarchies of layers.

Second, this ServiceGeoserver implementation is configurable. In other words, if the Service administrator intends to only make use (for the moment) of WFS functionality, they can customize this Service using the following definition.

providers:
  RemoteGeoServer:
    # minimal configuration requirements (where the real `GeoSever` service resides)
    # other optional parameters from `providers.cfg` can also be provided
    url: http://localhost:1234
    type: geoserver

    # customizable configuration (enable desired OWS/REST request handlers)
    # all OWS/REST services are enabled by default if no configuration is provided
    configuration:
      wfs: true
      wms: false
      wps: false
      api: false

This would make sure that request parsing and access to WMS, WPS and any REST API endpoints are disabled, but leaves the Resource definitions available for use at a later time if the administrator decides to eventually make use of them. For example, the administrator could decide to start using WMS as well without any further change needed other than updating this Service custom configuration and applying Permissions specific only to WMS. All other Applied Permissions to existing User, Group and Resource for that Service, as well as their full Resource tree hierarchy, would be automatically ported from the WFS to WMS request handlers.

Note

Custom configuration can be provided With either the providers.cfg (as presented above), in a Combined Configuration File as described in greater lengths within the Configuration chapter, or by providing the configuration field directly within the API request body during Service creation.

Service Synchronization

Added in version 0.7.

Some Service implementations offer a synchronization feature which is represented by field sync_type within Magpie API responses. When this parameter is defined (during Service creation, whether through API request or startup Configuration), the corresponding Service will be able to query the real remote service provider to retrieve actual Resource nested under it, based on the referenced implementation from the magpie.cli.sync_services module. Each of these synchronization implementations must derive from magpie.cli.sync_service.SyncServiceInterface and must populate the database with appropriate Resource types. This allows quick generation of the retrieved Resource tree hierarchy rather than manually creating each element.

Note

The depth of the Resource tree hierarchy that will be synchronized between Magpie and the remote service provider depends on the specific implementation of the sync_type referring to a derived magpie.cli.sync_service.SyncServiceInterface. These classes provide a parameter max_depth which can limit how many Resource must be generated. This is useful for entries that have very large and deeply nested structure that would take too long to synchronize. By default, max_depth is not set to pull the whole tree hierarchy.

Note

If only Scope.RECURSIVE Permission are being applied on the Service or their children Resource, it is better to enter fewer children element in the tree to reduce computation time of Effective Permissions. The complete hierarchy should be employed only when the depth of the tree is relatively shallow or that Scope.MATCH must be applied specifically for some Resource to obtain desired access behaviour.

When using the Magpie Docker image, the default command run the magpie-cron utility in parallel to the API. This cron job will periodically execute the Resource auto-synchronization feature for a given Service that supports it.

The synchronization mechanism can be launched from Magpie UI using the Sync button located on relevant pages.

See also

Utility magpie_sync_resources in Magpie CLI Helpers is also available to manually launch a Resource synchronization operation from the command line for supporting Service-types. This is the same operation that gets executed by magpie-cron.