magpie.cli.sync_resources ========================= .. py:module:: magpie.cli.sync_resources .. autoapi-nested-parse:: Synchronize local and remote resources. To implement a new service, see :class:`magpie.cli.sync_services.SyncServiceInterface`. .. seealso:: - :py:mod:`magpie.cli.sync_services` Attributes ---------- .. autoapisummary:: magpie.cli.sync_resources.SYNC_SERVICES_TYPES magpie.cli.sync_resources.LOGGER magpie.cli.sync_resources.CRON_SERVICE magpie.cli.sync_resources.OUT_OF_SYNC Classes ------- .. autoapisummary:: magpie.cli.sync_resources.SyncServiceDefault Functions --------- .. autoapisummary:: magpie.cli.sync_resources.get_resource_children magpie.cli.sync_resources.is_valid_resource_schema magpie.cli.sync_resources.make_logging_options magpie.cli.sync_resources.setup_logger_from_options magpie.cli.sync_resources.get_logger magpie.cli.sync_resources.merge_local_and_remote_resources magpie.cli.sync_resources._merge_resources magpie.cli.sync_resources._sort_resources magpie.cli.sync_resources._ensure_sync_info_exists magpie.cli.sync_resources._get_remote_resources magpie.cli.sync_resources._delete_records magpie.cli.sync_resources._create_main_resource magpie.cli.sync_resources._update_db magpie.cli.sync_resources._format_resource_tree magpie.cli.sync_resources._query_remote_resources_in_database magpie.cli.sync_resources.get_last_sync magpie.cli.sync_resources.fetch_all_services_by_type magpie.cli.sync_resources.fetch_single_service magpie.cli.sync_resources.fetch magpie.cli.sync_resources.setup_cron_logger magpie.cli.sync_resources.make_parser magpie.cli.sync_resources.main Module Contents --------------- .. py:function:: get_resource_children(resource: magpie.typedefs.ServiceOrResourceType, db_session: sqlalchemy.orm.session.Session, tree_service_builder: Optional[ziggurat_foundations.models.services.resource_tree.ResourceTreeService] = None, limit_depth: Optional[int] = None) -> magpie.typedefs.NestedResourceNodes Obtains the children resource node structure of the input service or resource. :param resource: Initial resource where to start building the tree from. :param db_session: Database connection to retrieve resources. :param tree_service_builder: Utility that build the tree (default: :py:data:`models.RESOURCE_TREE_SERVICE`). :param limit_depth: Maximum depth to look for children resources (very deep if not specified, could be slow). :returns: ``{node: Resource, children: {node_id: }}`` .. py:data:: SYNC_SERVICES_TYPES :type: Dict[magpie.typedefs.Str, Type[SyncServiceInterface]] .. py:class:: SyncServiceDefault(service_name, url) Bases: :py:obj:`SyncServiceInterface` .. py:property:: max_depth :type: None The max depth at which remote resources are fetched. .. py:method:: get_resources() -> magpie.typedefs.ServiceResourceTypeTree This is the function actually fetching the data from the remote service. Implement this for every specific service. :return: The returned dictionary must be validated by 'is_valid_resource_schema' .. py:function:: is_valid_resource_schema(resources: magpie.typedefs.AnyNestedChildrenTree) -> bool Validates the resource structure. Expected dictionary is a tree of the following form: .. code-block:: json { "resource_name_1": { "children": { "resource_name_3": {"children": {}}, "resource_name_4": {"children": {}} } } "resource_name_2": {"children": {}} } .. py:function:: make_logging_options(parser: argparse.ArgumentParser) -> None Defines argument parser options for logging operations. .. py:function:: setup_logger_from_options(logger: logging.Logger, args: argparse.Namespace) -> None Uses argument parser options to setup logging level from specified flags. Setup both the specific CLI logger that is provided and the generic `magpie` logger. .. py:function:: get_logger(name: magpie.typedefs.Str, level: Optional[int] = None, force_stdout: bool = None, message_format: Optional[magpie.typedefs.Str] = None, datetime_format: Optional[magpie.typedefs.Str] = None) -> logging.Logger Immediately sets the logger level to avoid duplicate log outputs from the `root logger` and `this logger` when `level` is ``logging.NOTSET``. .. py:data:: LOGGER .. py:data:: CRON_SERVICE :value: False .. py:data:: OUT_OF_SYNC .. py:function:: merge_local_and_remote_resources(resources_local: magpie.typedefs.RemoteResourceTree, service_sync_type: magpie.typedefs.Str, service_id: int, session: sqlalchemy.orm.session.Session) -> magpie.typedefs.RemoteResourceTree Main function to sync resources with remote server. .. py:function:: _merge_resources(resources_local: magpie.typedefs.RemoteResourceTree, resources_remote: magpie.typedefs.RemoteResourceTree, max_depth: Optional[int] = None) -> magpie.typedefs.RemoteResourceTree Merge resources_local and resources_remote, adding the following keys to the output: - remote_id: id of the RemoteResource - matches_remote: True or False depending if the resource is present on the remote server :returns: Dictionary tree of the merged resources. .. py:function:: _sort_resources(resources: magpie.typedefs.RemoteResourceTree) -> None Sorts a nested resource dictionary. The dictionary is expected to be valid as per :func:`sync_services.is_valid_resource_schema`. :return: None. Inplace modification. .. py:function:: _ensure_sync_info_exists(service_resource_id: int, session: sqlalchemy.orm.session.Session) -> None Make sure the RemoteResourcesSyncInfo entry exists in the database. .. py:function:: _get_remote_resources(service: magpie.models.Service) -> magpie.typedefs.JSON Request remote resources, depending on service type. :param service: service for which to fetch remove children resources :return: nested content as ``{"node_name": {"children": JSON, "resource_type": "resource_type"}}`` .. py:function:: _delete_records(service_id, session) Delete all RemoteResource based on a Service.resource_id. .. py:function:: _create_main_resource(service_id: int, session: sqlalchemy.orm.session.Session) -> None Creates a main resource for a service, whether one currently exists or not. Each RemoteResourcesSyncInfo has a main RemoteResource of the same name as the service. This is similar to the Service and Resource relationship. .. py:function:: _update_db(remote_resources: magpie.typedefs.RemoteResourceTree, service_id: int, session: sqlalchemy.orm.session.Session) -> None Writes remote resources to database. .. py:function:: _format_resource_tree(children: magpie.typedefs.ServiceResourceNodeTree) -> magpie.typedefs.RemoteResourceTree .. py:function:: _query_remote_resources_in_database(service_id: int, session: sqlalchemy.orm.session.Session) -> magpie.typedefs.RemoteResourceTree Reads remote resources from the RemoteResources table. No external request is made. :return: Dictionary of the form defined in :func:`sync_services.is_valid_resource_schema`. .. py:function:: get_last_sync(service_id: int, session: sqlalchemy.orm.session.Session) -> Optional[datetime.datetime] Obtain the date-time of the last known sync event for a service. .. py:function:: fetch_all_services_by_type(service_type: magpie.typedefs.Str, session: sqlalchemy.orm.session.Session) -> None Get remote resources for all services of a certain type. :param service_type: Service type for which all corresponding services will be synchronized. :param session: Database connexion to apply synchronization changes. .. py:function:: fetch_single_service(service: Union[magpie.models.Service, int], session: sqlalchemy.orm.session.Session) -> None Get remote resources for a single service. :param service: Specific service for which to synchronize remote resources. :param session: Database connexion to apply synchronization changes. .. py:function:: fetch(settings: Optional[magpie.typedefs.SettingsType] = None) -> None Main function to get all remote resources for each service and write to database. .. py:function:: setup_cron_logger(log_level: Union[magpie.typedefs.Str, int] = logging.INFO) -> None .. py:function:: make_parser() -> argparse.ArgumentParser .. py:function:: main(args: Optional[Sequence[magpie.typedefs.Str]] = None, parser: Optional[argparse.ArgumentParser] = None, namespace: Optional[argparse.Namespace] = None) -> Any Main entry point for cron service.