reorganize push_data_to_ccos and improve error handling

Author: TimidRobot

Pipfile

@@ -10,6 +10,9 @@ ipython = "*"
 isort = "*"
 
 [packages]
+asana = "*"
+emoji = "*"
+GitPython = "*"
 PyGithub = "*"
 PyYAML = "*"
 

Pipfile.lock

@@ -4,56 +4,87 @@ These are scripts used to maintain various pieces of CC's open source community
 infrastructure.
 
 
-## Scripts
+## Workflows
+
 
-###  `normalize_repos.py`
+###  Normalize Repos
 
+- Workflow
+  - Status: [![Normalize Repos][norm_badge]][norm_link]
+  - Schedule: Hourly at 45 minutes past the hour (`**:45`)
+  - YAML: [`normalize_repos.yml`][norm_pr_yml]
 - Script:  `normalize_repos.py`
   - Common Modules: [`ccos/`](ccos/)
   - Specific Modules: [`ccos/norm/`](ccos/norm/)
-- Workflow
-  - Status: [![Add Community PRs to Project][b1]][l1]
-  - Schedule: Hourly at 45 minutes past the hour
-  - YAML: [`normalize_repos.yml`][norm_pr_yml]
 - Required Environment Variables:
   - `ADMIN_GITHUB_TOKEN`
 
-This script ensures that all active repositories in the creativecommons GitHub
-organization are consistent in the following ways:
+This workflow ensures that all active repositories in the creativecommons
+GitHub organization are consistent in the following ways:
 - They have all the labels defined in `labels.yml` present.
 - They have standard branch protections set up (with some exceptions).
 
 This script will only update color and description of existing labels or create
 new labels. It will never delete labels.
 
 [norm_pr_yml]: .github/workflows/normalize_repos.yml
-[b2]: https://github.com/creativecommons/ccos-scripts/actions/workflows/normalize_repos.yml/badge.svg
-[l2]: https://github.com/creativecommons/ccos-scripts/actions/workflows/normalize_repos.yml
+[norm_badge]: https://github.com/creativecommons/ccos-scripts/actions/workflows/normalize_repos.yml/badge.svg
+[norm_link]: https://github.com/creativecommons/ccos-scripts/actions/workflows/normalize_repos.yml
 
 
-## Environment Variables
+### Push data to CC Open Source
+
+- Workflow
+  - Status: [![Push data to CC Open Source][data_badge]][data_link]
+  - Schedule: Daily at midnight:15 (`00:15`)
+  - YAML: [`push_data_to_ccos.yml`][push_ccos_yml]
+- Script:  `push_data_to_ccos.py`
+  - Common Modules: [`ccos/`](ccos/)
+  - Specific Modules: [`ccos/data/`](ccos/data/)
+- Required Environment Variables:
+  - `ADMIN_ASANA_TOKEN`
+  - `ADMIN_GITHUB_TOKEN`
 
+This workflow retreives data from Asana, formats it as a lektor databag, and
+pushes it to CC Open Source website source repository:
+- Data Source: [Community Team Tracking - Asana][asana] (limited access)
+- Data Destination:
+  - [`creativecommons.github.io-source`][ccos_source]
+    - [`databags/community_team_members.json`][db_community]
+    - [`databags/repos.json`][db_repos]
 
-### `ADMIN_GITHUB_TOKEN`
+The destination data is used by the following pages:
+- [Community Team Members — Creative Commons Open Source][ctlistpage]
+- [Open Source Projects — Creative Commons Open Source][osproj]
 
-GitHub token with admin permissions to the `creativecommons` GitHub
-organization.
+[data_badge]: https://github.com/creativecommons/ccos-scripts/actions/workflows/push_data_to_ccos.yml/badge.svg
+[data_link]: https://github.com/creativecommons/ccos-scripts/actions/workflows/push_data_to_ccos.yml
+[ctlistpage]: https://opensource.creativecommons.org/community/community-team/members/
+[osproj]: https://opensource.creativecommons.org/contributing-code/projects/
+[asana]: https://app.asana.com/0/1172465506923657/list
+[ccos_source]: https://github.com/creativecommons/creativecommons.github.io-source
+[db_community]: https://github.com/creativecommons/creativecommons.github.io-source/blob/main/databags/community_team_members.json
+[db_repos]: https://github.com/creativecommons/creativecommons.github.io-source/blob/main/databags/repos.json
 
 
-## Workflows
+## Environment Variables
+
+- `ADMIN_ASANA_TOKEN`: Asana token with access to the Creative Commons Asana
+  organization
+- `ADMIN_GITHUB_TOKEN`: GitHub token with admin permissions to the
+  `creativecommons` GitHub organization
+
+
+## Old Workflows Documentation
 
 | Workflow Name/Status | YML File Name | Workflow Purpose |
 | -------------------- | ------------- | ---------------- |
 | [![Add Community PRs to Project][b1]][l1] | [`add_community_pr.yml`][community_pr_yml] | Runs hourly at 5 minutes past every hour UTC and adds new Vocabulary issues to [Vocabulary: In Progress][vocab_in_progress] |
-| [![Push data to CC Open Source][b3]][l3] | [`push_data_to_ccos.yml`][push_ccos_yml] | Runs daily at 00:00 UTC and whenever someone pushes to the main branch and uses [`push_data_to_ccos`][push_to_ccos] |
 | [![Sync Community Teams with GitHub][b4]][l4] | [`sync_community_team.yml`][sync_team_yml] | Runs daily at 00:30 UTC and whenever someone pushes to the main branch and uses [`sync_community_team`][sync_team] |
 | [![Track new issues in backlog][b5]][l5] | [`track_backlog.yml`][track_backlog] | Runs hourly at 45 minutes past every hour UTC and adds PRs to [Active Sprint: Code Review][active_sprint] and new issues to [Backlog: Pending Review][backlog_pending]. Uses [dhruvkb/issue-projector][issue-projector]. |
 
 [b1]: https://github.com/creativecommons/ccos-scripts/actions/workflows/add_community_pr.yml/badge.svg
 [l1]: https://github.com/creativecommons/ccos-scripts/actions/workflows/add_community_pr.yml
-
-[b3]: https://github.com/creativecommons/ccos-scripts/actions/workflows/push_data_to_ccos.yml/badge.svg
-[l3]: https://github.com/creativecommons/ccos-scripts/actions/workflows/push_data_to_ccos.yml
 [b4]: https://github.com/creativecommons/ccos-scripts/actions/workflows/sync_community_team.yml/badge.svg
 [l4]: https://github.com/creativecommons/ccos-scripts/actions/workflows/sync_community_team.yml
 [b5]: https://github.com/creativecommons/ccos-scripts/actions/workflows/track_backlog.yml/badge.svg

README.md

@@ -6,24 +6,46 @@
 """
 
 # Standard library
+import inspect
 import logging
 import os
+import sys
 
 # Third-party
 import asana
 
 # First-party/Local
-import log
+from ccos import log
 
-ASANA_CLIENT = asana.Client.access_token(os.environ["ADMIN_ASANA_TOKEN"])
+ASANA_WORKSPACE_GID = "133733285600979"
 ASANA_PROJECT_GID = "1172465506923661"
 
-log.set_up_logging()
-logger = logging.getLogger("push_data_to_ccos")
+log_name = os.path.basename(os.path.splitext(inspect.stack()[-1].filename)[0])
+logger = logging.getLogger(log_name)
 log.reset_handler()
 
 
-def generate_databag():
+def setup_asana_client():
+    logger.log(logging.INFO, "Setting up Asana client...")
+    try:
+        asana_token = os.environ["ADMIN_ASANA_TOKEN"]
+    except KeyError:
+        logger.critical("missin ADMIN_ASANA_TOKEN environment variable")
+        sys.exit(1)
+    asana_client = asana.Client.access_token(asana_token)
+    try:
+        # Perform simple API operation to test authentication
+        asana_client.workspaces.get_workspace(ASANA_WORKSPACE_GID)
+    except asana.error.NoAuthorizationError as e:
+        logger.critical(
+            f"{e.status} {e.message} (is ADMIN_ASANA_TOKEN valid?)"
+        )
+        sys.exit(1)
+    logger.log(logging.INFO, "done.")
+    return asana_client
+
+
+def generate_databag(asana_client):
     """
     This method pulls the team members from Asana and
     loads them into the databag after a little
@@ -57,10 +79,11 @@ def generate_databag():
         ]
     }
     """
+
     logger.log(logging.INFO, "Pulling from Asana and generating databag...")
     databag = {"projects": [], "community_builders": []}
 
-    members = ASANA_CLIENT.tasks.find_by_section(
+    members = asana_client.tasks.find_by_section(
         ASANA_PROJECT_GID, opt_fields=["name", "custom_fields"]
     )
     logger.log(logging.INFO, "Team members pulled.")
@@ -164,5 +187,5 @@ def get_custom_field(task, field_name):
                 return field["text_value"]
 
 
-def get_community_team_data():
-    return prune_databag(sort_databag(generate_databag()))
+def get_community_team_data(asana_client):
+    return prune_databag(sort_databag(generate_databag(asana_client)))

push_data_to_ccos/get_community_team_data.py renamed to ccos/data/get_community_team_data.py

@@ -2,7 +2,9 @@
 # vim: set fileencoding=utf-8:
 
 # Standard library
+import inspect
 import logging
+import os.path
 
 # Third-party
 import emoji
@@ -11,13 +13,13 @@
 from github.GithubException import GithubException, UnknownObjectException
 
 # First-party/Local
-import log
-from push_data_via_git import GITHUB_ORGANIZATION, GITHUB_TOKEN
+from ccos import log
+from ccos.data.push_data_via_git import GITHUB_ORGANIZATION, GITHUB_TOKEN
 
 CC_METADATA_FILE_NAME = ".cc-metadata.yml"
 
-log.set_up_logging()
-logger = logging.getLogger("push_data_to_ccos")
+log_name = os.path.basename(os.path.splitext(inspect.stack()[-1].filename)[0])
+logger = logging.getLogger(log_name)
 log.reset_handler()
 
 

push_data_to_ccos/get_repo_data.py renamed to ccos/data/get_repo_data.py

@@ -2,6 +2,7 @@
 # vim: set fileencoding=utf-8:
 
 # Standard library
+import inspect
 import json
 import logging
 import os
@@ -10,7 +11,7 @@
 import git
 
 # First-party/Local
-import log
+from ccos import log
 
 GIT_USER_NAME = "CC creativecommons.github.io Bot"
 GIT_USER_EMAIL = "cc-creativecommons-github-io-bot@creativecommons.org"
@@ -29,8 +30,8 @@
 GIT_WORKING_DIRECTORY = f"{WORKING_DIRECTORY}/{GITHUB_REPO_NAME}"
 JSON_FILE_DIRECTORY = f"{GIT_WORKING_DIRECTORY}/databags"
 
-log.set_up_logging()
-logger = logging.getLogger("push_data_to_ccos")
+log_name = os.path.basename(os.path.splitext(inspect.stack()[-1].filename)[0])
+logger = logging.getLogger(log_name)
 log.reset_handler()