Services¶
This section describes Service and underlying Resource elements packaged with Magpie, as well as their respective functionalities and intended procedural behavior. Section Basic Service and Resource details first presents the general details to implement all Service implementations, followed by section Available Services which details specific implementations.
Basic Service and Resource details¶
Each Service is a specialization of a Resource (see: magpie.models.Service implementation).
Therefore, they can be listed and searched for either using /services API routes (using service_name) or using
/resources API routes (with their attributed resource_id), whichever is more convenient for the required needs.
On top of any Resource’s metadata, a Service provides specific information about its location, its
remote synchronization method (if any), and its exposed endpoint. Another important detail about the Service
is its type. This will not only dictate its purpose, but also define the whole schema of allowed Resource
under it (if any), as well as every one of their Allowed Permissions.
The final distinction between a Service and generic Resource is their position in the hierarchy. Only
Service-specialized Resource (literally resource_type = "service") are allowed to be placed at the
top of the tree hierarchy. All generic Resource nodes must be nested under some root Service. Relative
references are indicated using parent_id and root_service_id in corresponding Resource details.
Every Service type provided by Magpie must derive from magpie.services.ServiceInterface. Each
specific implementation (see Available Services) serves to convert a given incoming HTTP request components
(method, path, query parameters, body, etc.) into the appropriate Service, Resource and
Permission elements. This ultimately provides the required elements to resolve ACL access of a
Request User toward the targeted Resource according to its
Effective Permissions.
In order to implement a new Service type, two (2) methods and a few attributes are required. The first method is
magpie.services.ServiceInterface.permission_requested() which basically indicates how the HTTP request should be
interpreted into a given Permission. The second is magpie.services.ServiceInterface.resource_requested()
which similarly tells the interpretation method to convert the request into a magpie.models.Resource reference.
Whenever Effective Permission or ACL needs to be resolved in order to determine if a Request User can have access or not to a Resource, Magpie will employ the appropriate Service implementation and call the methods to process the result.
Changed in version 3.0: All the resolution of Access and Scope
modifiers are automatically handled according to the applied Permission on the Resource hierarchy.
Therefore, no specific action is required to support these features for new Service implementations.
See Permission Definition and Modifiers for details.
On top of the above methods, the following attributes must be defined.
Attribute |
Description |
|---|---|
Defines the mapping of registered Service to the appropriate implementation type. Each implementation must have an unique value. |
|
|
Defines the Allowed Permissions that can be applied onto the Service reference itself. |
|
Map of the allowed children Resource under the Service and their corresponding
Allowed Permissions for each case. Leaving this map empty will disallow the
creation of any children Resource, making the Service the unique applicable element of the
hierarchy. Note that each Resource implemented by a derived class of |
|
Map of allowed Resource type nesting hierarchy for the Service. This controls whether some children Resource can be placed under another to limit creation only to cases that are relevant for the implemented Service. |
|
Parameter that controls whether the Service supports custom configuration, providing means to slightly alter, enable or disable certain behaviours of the parsing methodology of request into Resource and Permission components for the given Service. |
See also
ServiceTHREDDS and ServiceGeoserver for examples of Service implementations that support custom configuration.
On top of the above definitions, any service that derives from magpie.services.ServiceOWS should also provide
follow parameters for parsing OWS requests.
Attribute |
Description |
|---|---|
|
Represents specific parameter names that can be preprocessed during HTTP request parsing to ease following resolution of ACL use cases. |
|
Provide the reference OWS type for handling requests with proper parsers and resolvers. |
Furthermore, some Services specifically implement extended OWS utilities offered by
GeoServer. They derive from ServiceGeoserverBase and should provide the following additional parameters.
Attribute |
Description |
|---|---|
Indicates if the Service is allowed to employ scoped |
|
Indicates if the Service supports multiple simultaneous Resource references within a single request (see Permissions Resolution for more details`), which must all be considered for Effective Resolution. |
|
|
Name of one or many request query parameter(s) from which to extract multiple equivalent Resource references when the Service supports multiple representations or notation conventions. |
Changed in version 3.21: - Attribute ServiceOWS.params_expected has been moved from ServiceInterface to instead be directly
under ServiceOWS since it applies only to derived classes from that base.
Added in version 3.21: - Attribute ServiceInterface.child_structure_allowed
- Attribute ServiceInterface.configurable
- Attribute ServiceOWS.service_base
- Attribute ServiceGeoserverBase.resource_scoped
- Attribute ServiceGeoserverBase.resource_multi
- Attribute ServiceGeoserverBase.resource_param
Available Services¶
See also
Module magpie.services contains the implementation of every Service presented below.
See also
Multiple of the following implementation details require comprehension of the concepts introduced in Permission Definition and Modifiers. Consider looking this up if descriptions feel hard to understand at first glance.
ServiceAccess¶
The implementation of this Service is handled by class magpie.services.ServiceAccess. It is intended to
act as a simple all-or-nothing endpoint barrier, where only Permission.ACCESS can be applied, and only
directly on the Service itself. A User or Group that has that Permission with
Access.ALLOW will be able to reach the Service. Any other operation (or explicit Access.DENY)
will all result into denied access to the private URL registered by the Service.
Changed in version 3.0: This Service implementation dates prior to the integration of Access and Scope concepts
that now provides modifiers to the Permission resolution methodology, notably prior to the addition of
Access.DENY. This Service is equivalent to a simplified version of the fine-grained ServiceAPI
alternative where no children Resource would be registered.
Note
Due to the single-level aspect of this Service, the Scope Permission modifier does not
have any effect.
ServiceAPI¶
The implementation of this Service is handled by class magpie.services.ServiceAPI. It refers to a
remote URL endpoint that should have a Resource tree formed out of the path segments. The Service only
has one (1) type of Resource, namely magpie.models.Route, that can have an unlimited amount of nested
children of the same type. The Allowed Permissions for this Service
are Permission.READ and Permission.WRITE. All requests using GET or HEAD are mapped
to Permission.READ access while all other HTTP methods represent a Permission.WRITE access.
Request path segments follow the natural hierarchy of the nested magpie.models.Route under the Service.
For example, a proxy employing magpie.adapter.MagpieAdapter such that {PROXY_URL} is the base of the
request path and the magpie.services.ServiceAPI named SomeAPI was registered would be mapped against
each sub-Resource as presented below.
{PROXY_URL}/SomeAPI/some-resource/child_resource/final
{PROXY_URL}/SomeAPI/other-resource/subResource
SomeAPI [service: ServiceAPI]
some-resource [resource: Route]
child_resource [resource: Route]
final [resource: Route]
other-resource [resource: Route]
subResource [resource: Route]
Every magpie.models.Route as well as the Service itself can have the Permission based on the
HTTP method of the incoming request. All Access and Scope modifiers are also supported for highly
customizable ACL combinations. See Permission Definition and Modifiers for further details.
ServiceTHREDDS¶
The implementation of this Service is handled by class magpie.services.ServiceTHREDDS. It refers to a
remote data server named Thematic Real-time Environmental Distributed Data Services (THREDDS). The Service
employs two (2) types of Resource, namely magpie.models.Directory and magpie.models.File.
All the directory resources can be nested any number of times, and files can only reside as leaves of the hierarchy,
similarly to a traditional file system. The Allowed Permissions on both the Service
itself or any of its children Resource are Permission.BROWSE, Permission.READ, and
Permission.WRITE (see note below regarding this last permission).
Added in version 3.1: The Permission.BROWSE permission is used to provide listing access of contents when targeting a
Resource of type magpie.models.Directory. When targeting a magpie.models.File, it instead
provides metadata access to that file.
Permission Permission.READ can be applied to all of the resources, but will only effectively make sense when
attempting access of a specific Resource of type magpie.models.File.
Changed in version 3.1: Permission Permission.READ does not offer metadata content listing of magpie.models.Directory
anymore. For this, Permission.BROWSE should be used instead. Setting Permission.READ on a
directory will only be logical when combined with Scope.RECURSIVE, in which case Magpie will interpret
the Effective Permission to allow read access to all magpie.models.File under that directory, at
any depth level, unless denied by a lower-level specification.
Finally, Permission.WRITE can also be applied on all of the resources, but are not explicitly employed during
parsing of incoming requests.
Note
The Permission.WRITE is not handled by ServiceTHREDDS itself during ACL resolution as it is not
considered by magpie.services.ServiceTHREDDS.permission_requested() that never returns this value. The method
only returns either Permission.BROWSE or Permission.READ. A User or Group can still
have this Applied Permission to allow a third party service to interrogate Magpie API about the presence
of Permission.WRITE permission and perform the appropriate action with the result. The
Effective Permission API routes will provide the resolved access. It is only Twitcher proxy that
will not be able to make use of it during incoming requests as it depends on
magpie.adapter.magpieowssecurity.MagpieOWSSecurity, which in turn employs the result from the ACL.
The Permission.WRITE is mostly preserved for backward compatibility of services that employed
ServiceTHREDDS to obtain information about which directory or files (which registered Magpie Resource)
are writable or not, although using another upload methodology that is not specifically executed via the actual
remote THREDDS service.
As presented above, the main two permissions are Permission.BROWSE and Permission.READ which
correspondingly serve to retrieve metadata and actual data of a given Resource. To distinguish requests
between these two types of contents, ServiceTHREDDS employs two parts from the request path, the sub-path prefix
and the file extension. The default methodology employed to categorize these two types of content is represented
by the below configuration.
Note
A custom categorization between metadata and data contents can be provided With either the providers.cfg or a Combined Configuration File as described in greater lengths within the Configuration chapter.
providers:
LocalThredds:
# minimal configuration requirements (where the real `THREDDS` service resides)
# other optional parameters from `providers.cfg` can also be provided
url: http://localhost:1234
type: thredds
# customizable request parsing methodology (specific to `thredds` type)
configuration:
# path prefix to skip (strip) before processing the rest of the path in case the
# registered service URL in Magpie does not have the same root as proxied by Twitcher public URL
skip_prefix: thredds
# define which pattern matches that will map different path variations into same file resources
# this can be used to consider two file extensions as the same resource to avoid duplication of permissions
file_patterns:
# note: make sure to employ quotes and double escapes to avoid parsing YAML error
# patterns are **NOT** UNIX filters, but regex format (eg: dot is 'any-character', not a literal dot)
- ".*\\.nc"
# path prefix to resources to be considered as BROWSE-able metadata (directory listing or file details)
metadata_type:
prefixes:
- null # note: special YAML value evaluated as `no-prefix`, use quotes if literal value is needed
- "catalog\\.\\w+" # note: special case for `THREDDS` top-level directory (root) accessed for `BROWSE`
- catalog
- ncml
- uddc
- iso
# path prefix to resources to be considered as READ-able data (i.e.: file contents)
data_type:
prefixes:
- fileServer
- dodsC
- dap4
- wcs
- wms
Warning
Regular Expression Patterns
Ensure to properly escape special characters, notably the dot (.), to avoid granting unexpected permissions that
would match any character. Format employed in above patterns are traditional Regex, not UNIX style filters.
Changed in version 3.2: Added catalog specific patterns by default to metadata prefixes that composes another valid URL variant to
request Permission.BROWSE directly on the top-level THREDDS service (directory), although
<prefix_type> is otherwise always expected at that second position path segment after the service name
(see below example). The pattern allows multiple extensions to support the various representation modes of the
catalog listing (e.g.: XML, HTML, etc.).
As of that version, the prefixes entries also support patterns, using standard regular expression syntax.
Changed in version 3.3: Added skip_prefix to allow ignoring intermediate path segments between the service name and the desired
<prefix_type> position. A typical use case with THREDDS is the /thredds prefix it adds between its
API entrypoint and Tomcat service running it. If this feature is not needed, it can be disabled by setting the
parameter to null.
Assuming a proxy intended to receive incoming requests configured with magpie.adapter.MagpieAdapter such that
{PROXY_URL} is the base path, the following path would point toward the registered service with the above YAML
configuration.
{PROXY_URL}/LocalThredds
An incoming request will be parsed according to configured values against the following format.
{PROXY_URL}/LocalThredds[/skip/prefix]/<prefix_type>/.../<file>
The above template demonstrates that Magpie will attempt to match the <prefix_type> part of the request path with
any of the listed prefixes in the configuration (metadata or data). The <prefix_type> location in the path
(i.e.: which segment to consider as <prefix_type>) will be determined by the next part following configuration value
defined by skip_prefix (any number of sub-parts). If skip_prefix cannot be located in the request path or was
defined as null, the first part after the service name is simply assumed as the <prefix_type> to lookup.
If a match is found between the various prefixes and <prefix_type>, the corresponding metadata or data
content will be assumed, according to where the match entry was located, to determine whether the requested
Resource should be validated respectively for Permission.BROWSE or Permission.READ access.
If no <prefix_type> can be resolved, the Permission will be immediately assumed as Access.DENY
regardless of type. To allow top-level access directly on the Service’s root without <prefix_type>, it is
important to provide null within the desired prefixes list. Duplicates between the two lists of prefixes
will favor entries in metadata_type over data_type.
After resolution of the content type from <prefix_type>, the resolution of any amount of
magpie.models.Directory Resource will be attempted. Any missing children directory Resource
will terminate the lookup process immediately, and ACL will be resolved considering Scope.RECURSIVE of
any applicable parent Resource for the given Permission selected by <prefix_type> and from where
lookup stopped.
Once the last element of the path is reached, the file_patterns will be applied against <file> in order to
attempt extracting the targeted magpie.models.File Resource. Patterns are applied until the first
positive match is found. Therefore, order is important if providing multiple patterns. For example, if the path ended
with file.ncml and, that both .*\\.ncml and .*\\.nc where defined in the configuration in that specific
order, the result will first match .*\\.ncml, and the final magpie.models.File Resource value will
be considered as file.ncml for lookup. In this case, another request using only file.nc would lookup an entirely
different Resource, since the second pattern would be the first successful match. On the other hand, if only
.*\\.nc was defined in file_patterns, the matched pattern would convert both the names file.ncml and
file.nc to file.nc, which will lookup exactly the same magpie.models.File reference. The most common
usage of this feature is to support additional extension suffixes as the same file with regrouped permissions such that
file.nc, file.nc.ascii?, file.nc.html, etc. all correspond to single and common Resource.
To summarize, if file_patterns produces a match, that matched portion will be used as lookup value of the
Resource. Otherwise, if not any match could be found amongst all the file_patterns, the plain <file>
name is used directly (as is from the specified request path). The plain name is also used if file_patterns is
explicitly specified as an empty list or null. Not explicitly overriding the field will result into using the
above default file_patterns. The file_patterns allow for example to consider file.nc, file.ncml and
file.nc.html as the same Resource internally, which avoids duplicating Applied Permission across
multiple Resource for their corresponding metadata or data representations.
ServiceWFS¶
See also
ServiceGeoserverWFS for a GeoServer flavoured implementation with only WFS support.
Consider using ServiceGeoserver for multi-OWS implementation support under a common endpoint representing a GeoServer instance.
https://www.ogc.org/standards/wfs (OpenGIS WFS 2.0.0 implementation)
This implementation is defined by magpie.services.ServiceWFS.
It implements the original standard WFS definition. There is NO concept of magpie.models.Workspace
for this Service implementation. Features are accessed directly using magpie.models.Layer typed
Resources.
ServiceGeoserverWFS¶
See also
https://docs.geoserver.org/latest/en/user/services/wfs/reference.html
Consider using ServiceGeoserver for multi-OWS implementation support under a common endpoint representing a GeoServer instance.
This implementation is defined by magpie.services.ServiceGeoserverWFS.
It implements some extensions to base WFS by providing more Permission and scoping of
magpie.models.Layer under magpie.models.Workspace in a file-system-like fashion.
ServiceBaseWMS¶
See also
Derived implementations: - ServiceGeoserverWMS - ServiceNCWMS2
This is a partial base class employed to represent OWS Web Map Service extended via other complete classes. It cannot be employed directly as Service instance. The derived classes provide different parsing methodologies and children Resource representation according to their respective functionalities.
It provides support for the following permissions, each corresponding to the appropriate functionality of WMS:
Similar to any other OWS based Service, the HTTP request takes a request query parameter that
indicates which of the above Permission is being requested.
ServiceGeoserverWMS¶
See also
Base class: ServiceBaseWMS
https://docs.geoserver.org/latest/en/user/services/wms/reference.html
Consider using ServiceGeoserver for multi-OWS implementation support under a common endpoint representing a GeoServer instance.
This implementation is defined by magpie.services.ServiceGeoserverWMS. It extends the base class by using
children Resource defined by magpie.models.Workspace, which supports the same set of Permission
as their parent Service. Each of those magpie.models.Workspace correspond to the equivalent element
provided to GeoServer based HTTP request using query parameter layers, following format
layers=<Workspace>:<LayerName>. The Permission is obtained from the request query parameter.
ServiceNCWMS2¶
See also
This implementation is defined by magpie.services.ServiceNCWMS2. It extends the base class by using
children Resource defined as magpie.models.Directory and magpie.models.File instances but,
using the corresponding Permission entries from ServiceBaseWMS class instead of the default
Permission.READ and Permission.WRITE (i.e.: see ServiceTHREDDS). The general idea is that the remote
ncWMS2 service provider being represented by this Service points to the same NetCDF file resources as
offered by THREDDS, but for mapping display. The HTTP request therefore points toward another proxy endpoint and
employs different query parameters specific to WMS requests (instead of THREDDS), although the provided file
reference is technically the same. For this reason, the same Resource hierarchy is supported, with any number
of nested magpie.models.Directory and magpie.models.File as leaves. The targeted Resource by
the HTTP request is extracted from either the dataset, layername or layers query parameter formatted as
relative file path from the THREDDS` root. The applicable query parameter depends on the appropriate
:term:`Permission` being requested based on the provided ``request query parameter.
Note
Although the class name employs NCWMS2, the registered type is represented by the string ncwms for
executing requests toward the Magpie API and contents returned in its responses.
ServiceWPS¶
See also
Consider using ServiceGeoserver for multi-OWS implementation support under a common endpoint representing a GeoServer instance.
The implementation of this Service is handled by class magpie.services.ServiceWPS. It is intended to
control access to the operations provided by an OWS Web Processing Service. This Service allows
one (1) type of child Resource, namely the magpie.models.Process which represent the execution units
that are registered under a remote WPS. Every magpie.models.Process cannot itself have a child
Resource, making ServiceWPS maximally a 2-tier level hierarchy.
There are three (3) types of Allowed Permission which each represent an operation that can be requested from it
(via request query parameter value of the HTTP request), specifically the Permission.GET_CAPABILITIES,
Permission.DESCRIBE_PROCESS, and Permission.EXECUTE. The Permission.GET_CAPABILITIES
corresponds to the retrieval of available list of Processes on the WPS instance, and therefore, can only be applied
on the top-level Service. The other two permissions can be applied on either the Service or specifically
on individual magpie.models.Process Resource definitions. When applied to the Service with
Scope.RECURSIVE modifier, the corresponding Permission becomes effective to all underlying Processes.
Otherwise, the Permission applied on specific magpie.models.Process entries control the specific
ACL only for it. When a specific Permission is involved on a magpie.models.Process during
Effective Permission resolution, the value of the query parameter identifier is employ to attempt mapping
it against an existing Resource. The resolution of Effective Permissions in the
event of multi-level tree Resource is computed in the usual manner described in the Permissions chapter.
Warning
When applying permissions on a per-magpie.models.Process basis, Scope.MATCH modifier is recommended
if only a specific request should be granted access, although Scope.RECURSIVE would have the same effect
currently because these resources are necessarily leaf nodes by definition. This is to prevent unexpectedly granting
additional lower-level Resource access in the event this definition gets modified or extended in the future.
ServiceGeoserverWPS¶
See also
Consider using ServiceGeoserver for multi-OWS implementation support under a common endpoint representing a GeoServer instance.
The implementation of this Service is handled by class magpie.services.ServiceGeoserverWPS.
It offers similar operations and request handling to ServiceWPS, but adds scoped magpie.models.Workspace
definition of magpie.models.Process as for other GeoServer base Services, in order to
properly parse request paths that include such references. Other than the Resource hierarchy being nested by
magpie.models.Workspace, the rest of the parsing and handling methodology is equivalent to ServiceWPS.
ServiceGeoserver¶
Added in version 3.21.
This Service is combined OWS implementation for GeoServer that allows simultaneous representation of
WFS, WMS and WPS Resources all nested under a single reference hosted under
common remote URL. Using this implementation, magpie.models.Workspace are first required as immediate children
under the root Service, and can be followed by both Resources of type
magpie.models.Layer and magpie.models.Process.
There are two main advantages of using this combined implementation over their specific OWS counterparts.
First, using the same Resources to represent corresponding elements in the GeoServer across
OWS endpoints reduces the chances of inconsistent access to otherwise equivalent Resources.
For example, a magpie.models.Layer that is granted Permission to retrieve features (WFS) or
to render their map raster (WMS) will be managed using the same reference. Using distinct
ServiceGeoserverWFS and ServiceGeoserverWMS Service in Magpie would require from to
administrator to always maintain their Permissions in sync. This is prone to many errors and
confusion when managing multiple large hierarchies of layers.
Second, this ServiceGeoserver implementation is configurable. In other words, if the Service administrator intends to only make use (for the moment) of WFS functionality, they can customize this Service using the following definition.
providers:
RemoteGeoServer:
# minimal configuration requirements (where the real `GeoSever` service resides)
# other optional parameters from `providers.cfg` can also be provided
url: http://localhost:1234
type: geoserver
# customizable configuration (enable desired OWS/REST request handlers)
# all OWS/REST services are enabled by default if no configuration is provided
configuration:
wfs: true
wms: false
wps: false
api: false
This would make sure that request parsing and access to WMS, WPS and any REST API endpoints are disabled, but leaves the Resource definitions available for use at a later time if the administrator decides to eventually make use of them. For example, the administrator could decide to start using WMS as well without any further change needed other than updating this Service custom configuration and applying Permissions specific only to WMS. All other Applied Permissions to existing User, Group and Resource for that Service, as well as their full Resource tree hierarchy, would be automatically ported from the WFS to WMS request handlers.
Note
Custom configuration can be provided With either the providers.cfg (as presented above), in
a Combined Configuration File as described in greater lengths within the Configuration chapter,
or by providing the configuration field directly within the API request body during
Service creation.
Service Synchronization¶
Added in version 0.7.
Some Service implementations offer a synchronization feature which is represented by field sync_type within
Magpie API responses. When this parameter is defined (during Service creation, whether through API request or
startup Configuration), the corresponding Service will be able to query the real
remote service provider to retrieve actual Resource nested under it, based on the referenced implementation
from the magpie.cli.sync_services module. Each of these synchronization implementations must derive from
magpie.cli.sync_service.SyncServiceInterface and must populate the database with appropriate Resource
types. This allows quick generation of the retrieved Resource tree hierarchy rather than manually creating each
element.
Note
The depth of the Resource tree hierarchy that will be synchronized between Magpie and the
remote service provider depends on the specific implementation of the sync_type referring to a derived
magpie.cli.sync_service.SyncServiceInterface. These classes provide a parameter max_depth which
can limit how many Resource must be generated. This is useful for entries that have very large and deeply
nested structure that would take too long to synchronize. By default, max_depth is not set to pull the whole
tree hierarchy.
Note
If only Scope.RECURSIVE Permission are being applied on the Service or their children
Resource, it is better to enter fewer children element in the tree to reduce computation time of
Effective Permissions. The complete hierarchy should be employed only when the depth
of the tree is relatively shallow or that Scope.MATCH must be applied specifically for some Resource
to obtain desired access behaviour.
When using the Magpie Docker image, the default command run the magpie-cron utility in parallel to the API. This cron job will periodically execute the Resource auto-synchronization feature for a given Service that supports it.
The synchronization mechanism can be launched from Magpie UI using the Sync button located on relevant pages.
See also
Utility magpie_sync_resources in Magpie CLI Helpers is also available to manually launch a
Resource synchronization operation from the command line for supporting Service-types.
This is the same operation that gets executed by magpie-cron.
