magpie.cli.sync_resources

Synchronize local and remote resources.

To implement a new service, see magpie.cli.sync_services.SyncServiceInterface.

See also

• magpie.cli.sync_services

Attributes

SYNC_SERVICES_TYPES
LOGGER
CRON_SERVICE
OUT_OF_SYNC

Classes

SyncServiceDefault

Functions

get_resource_children(...)	Obtains the children resource node structure of the input service or resource.
is_valid_resource_schema(→ bool)	Validates the resource structure.
make_logging_options(→ None)	Defines argument parser options for logging operations.
setup_logger_from_options(→ None)	Uses argument parser options to setup logging level from specified flags.
get_logger(→ logging.Logger)	Immediately sets the logger level to avoid duplicate log outputs from the root logger and this logger when
merge_local_and_remote_resources(...)	Main function to sync resources with remote server.
_merge_resources(→ magpie.typedefs.RemoteResourceTree)	Merge resources_local and resources_remote, adding the following keys to the output:
_sort_resources(→ None)	Sorts a nested resource dictionary.
_ensure_sync_info_exists(→ None)	Make sure the RemoteResourcesSyncInfo entry exists in the database.
_get_remote_resources(→ magpie.typedefs.JSON)	Request remote resources, depending on service type.
_delete_records(service_id, session)	Delete all RemoteResource based on a Service.resource_id.
_create_main_resource(→ None)	Creates a main resource for a service, whether one currently exists or not.
_update_db(→ None)	Writes remote resources to database.
_format_resource_tree(→ magpie.typedefs.RemoteResourceTree)
_query_remote_resources_in_database(...)	Reads remote resources from the RemoteResources table. No external request is made.
get_last_sync(→ Optional[datetime.datetime])	Obtain the date-time of the last known sync event for a service.
fetch_all_services_by_type(→ None)	Get remote resources for all services of a certain type.
fetch_single_service(→ None)	Get remote resources for a single service.
fetch(→ None)	Main function to get all remote resources for each service and write to database.
setup_cron_logger(→ None)
make_parser(→ argparse.ArgumentParser)
main(→ Any)	Main entry point for cron service.

Module Contents

magpie.cli.sync_resources.get_resource_children(resource: magpie.typedefs.ServiceOrResourceType, db_session: sqlalchemy.orm.session.Session, tree_service_builder: ziggurat_foundations.models.services.resource_tree.ResourceTreeService | None = None, limit_depth: int | None = None) → magpie.typedefs.NestedResourceNodes

Obtains the children resource node structure of the input service or resource.

Parameters:

• resource – Initial resource where to start building the tree from.

• db_session – Database connection to retrieve resources.

• tree_service_builder – Utility that build the tree (default: models.RESOURCE_TREE_SERVICE).

• limit_depth – Maximum depth to look for children resources (very deep if not specified, could be slow).

Returns:

{node: Resource, children: {node_id: <recursive>}}

magpie.cli.sync_resources.SYNC_SERVICES_TYPES: Dict[magpie.typedefs.Str, Type[SyncServiceInterface]]

class magpie.cli.sync_resources.SyncServiceDefault(service_name, url)

Bases: SyncServiceInterface

property max_depth: None

The max depth at which remote resources are fetched.

get_resources() → magpie.typedefs.ServiceResourceTypeTree

This is the function actually fetching the data from the remote service. Implement this for every specific service.

Returns:

The returned dictionary must be validated by ‘is_valid_resource_schema’

magpie.cli.sync_resources.is_valid_resource_schema(resources: magpie.typedefs.AnyNestedChildrenTree) → bool

Validates the resource structure.

Expected dictionary is a tree of the following form:

{
    "resource_name_1": {
        "children": {
            "resource_name_3": {"children": {}},
            "resource_name_4": {"children": {}}
        }
    }
    "resource_name_2": {"children": {}}
}

magpie.cli.sync_resources.make_logging_options(parser: argparse.ArgumentParser) → None

Defines argument parser options for logging operations.

magpie.cli.sync_resources.setup_logger_from_options(logger: logging.Logger, args: argparse.Namespace) → None

Uses argument parser options to setup logging level from specified flags.

Setup both the specific CLI logger that is provided and the generic magpie logger.

magpie.cli.sync_resources.get_logger(name: magpie.typedefs.Str, level: int | None = None, force_stdout: bool = None, message_format: magpie.typedefs.Str | None = None, datetime_format: magpie.typedefs.Str | None = None) → logging.Logger

Immediately sets the logger level to avoid duplicate log outputs from the root logger and this logger when level is logging.NOTSET.

magpie.cli.sync_resources.LOGGER

magpie.cli.sync_resources.CRON_SERVICE = False

magpie.cli.sync_resources.OUT_OF_SYNC

magpie.cli.sync_resources.merge_local_and_remote_resources(resources_local: magpie.typedefs.RemoteResourceTree, service_sync_type: magpie.typedefs.Str, service_id: int, session: sqlalchemy.orm.session.Session) → magpie.typedefs.RemoteResourceTree

Main function to sync resources with remote server.

magpie.cli.sync_resources._merge_resources(resources_local: magpie.typedefs.RemoteResourceTree, resources_remote: magpie.typedefs.RemoteResourceTree, max_depth: int | None = None) → magpie.typedefs.RemoteResourceTree

Merge resources_local and resources_remote, adding the following keys to the output:

• remote_id: id of the RemoteResource

• matches_remote: True or False depending if the resource is present on the remote server

Returns:

Dictionary tree of the merged resources.

magpie.cli.sync_resources._sort_resources(resources: magpie.typedefs.RemoteResourceTree) → None

Sorts a nested resource dictionary.

The dictionary is expected to be valid as per sync_services.is_valid_resource_schema().

Returns:

None. Inplace modification.

magpie.cli.sync_resources._ensure_sync_info_exists(service_resource_id: int, session: sqlalchemy.orm.session.Session) → None

Make sure the RemoteResourcesSyncInfo entry exists in the database.

magpie.cli.sync_resources._get_remote_resources(service: magpie.models.Service) → magpie.typedefs.JSON

Request remote resources, depending on service type.

Parameters:

service – service for which to fetch remove children resources

Returns:

nested content as {"node_name": {"children": JSON, "resource_type": "resource_type"}}

magpie.cli.sync_resources._delete_records(service_id, session)

Delete all RemoteResource based on a Service.resource_id.

magpie.cli.sync_resources._create_main_resource(service_id: int, session: sqlalchemy.orm.session.Session) → None

Creates a main resource for a service, whether one currently exists or not.

Each RemoteResourcesSyncInfo has a main RemoteResource of the same name as the service. This is similar to the Service and Resource relationship.

magpie.cli.sync_resources._update_db(remote_resources: magpie.typedefs.RemoteResourceTree, service_id: int, session: sqlalchemy.orm.session.Session) → None

Writes remote resources to database.

magpie.cli.sync_resources._format_resource_tree(children: magpie.typedefs.ServiceResourceNodeTree) → magpie.typedefs.RemoteResourceTree

magpie.cli.sync_resources._query_remote_resources_in_database(service_id: int, session: sqlalchemy.orm.session.Session) → magpie.typedefs.RemoteResourceTree

Reads remote resources from the RemoteResources table. No external request is made.

Returns:

Dictionary of the form defined in sync_services.is_valid_resource_schema().

magpie.cli.sync_resources.get_last_sync(service_id: int, session: sqlalchemy.orm.session.Session) → datetime.datetime | None

Obtain the date-time of the last known sync event for a service.

magpie.cli.sync_resources.fetch_all_services_by_type(service_type: magpie.typedefs.Str, session: sqlalchemy.orm.session.Session) → None

Get remote resources for all services of a certain type.

Parameters:

• service_type – Service type for which all corresponding services will be synchronized.

• session – Database connexion to apply synchronization changes.

magpie.cli.sync_resources.fetch_single_service(service: magpie.models.Service | int, session: sqlalchemy.orm.session.Session) → None

Get remote resources for a single service.

Parameters:

• service – Specific service for which to synchronize remote resources.

• session – Database connexion to apply synchronization changes.

magpie.cli.sync_resources.fetch(settings: magpie.typedefs.SettingsType | None = None) → None

Main function to get all remote resources for each service and write to database.

magpie.cli.sync_resources.setup_cron_logger(log_level: magpie.typedefs.Str | int = logging.INFO) → None

magpie.cli.sync_resources.make_parser() → argparse.ArgumentParser

magpie.cli.sync_resources.main(args: Sequence[magpie.typedefs.Str] | None = None, parser: argparse.ArgumentParser | None = None, namespace: argparse.Namespace | None = None) → Any

Main entry point for cron service.