Skip to content

Commit cc64fc8

Browse files
TimidRobotTimidRobot
authored
Merge pull request #516 from creativecommons/bin-scripts
Improve helper scripts
2 parents 4763afd + ac936a2 commit cc64fc8

20 files changed

+428
-180
lines changed

‎.github/workflows/check-python-versions.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,4 +16,4 @@ jobs:
1616
uses: actions/checkout@v4
1717

1818
- name: Run script to check Python versions
19-
run: ./dev/check_python_versions.sh
19+
run: ./bin/check_python_versions.sh

‎README.md

+55-17
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,7 @@ repository (where `manage.py` is).
129129
6. Initialize data.
130130
Open a separate terminal tab, and in the same directory, run:
131131
```shell
132-
./dev/init_data.sh
132+
./bin/init_data.sh
133133
```
134134
1. Deletes database (which may not yet exist)
135135
2. Initializes database
@@ -215,26 +215,52 @@ Repository](#data-repository), above).
215215
#### Static Files Process
216216
217217
This process will write the HTML files in the cc-legal-tools-data clone
218-
directory under `docs/`. It will not commit the changes (`--nogit`) and will
219-
not push any commits (`--nopush` is implied by `--nogit`).
218+
directory under `docs/`. By default it doesn't commit any changes.
220219

221220
1. Ensure the [Data Repository](#data-repository), above, is in place
222221
2. Ensure [Docker Compose Setup](#docker-compose-setup), above, is complete
223-
3. Delete the contents of the `docs/` directory and then recreate/copy the
224-
static files it should contain:
222+
3. Run the helper script:
225223
```shell
226-
docker compose exec app ./manage.py publish -v2
224+
./bin/publish.sh
227225
```
226+
- Example output:
227+
```
228+
# Django publish 19:36:20
229+
INFO 17:36:21 Checking legal code titles
230+
INFO 17:36:23 Purging output_dir: /home/cc/cc-legal-tools-data/docs
231+
INFO 17:36:31 Collecting static files
232+
66 static files copied to '/home/cc/cc-legal-tools-app/tmp/public/static', 158 unmodified.
233+
INFO 17:36:31 Writing robots.txt
234+
INFO 17:36:31 Copying WordPress content files
235+
INFO 17:36:31 Copying static cc-legal-tools files
236+
INFO 17:36:31 Copying static RDF/XML files
237+
INFO 17:36:31 Distilling index.rdf
238+
INFO 17:36:33 Distilling images.rdf
239+
INFO 17:36:33 Distilling ns.html
240+
INFO 17:36:33 Copying plaintext legal code
241+
INFO 17:36:33 Distilling dev index
242+
INFO 17:36:34 Distilling lists
243+
INFO 17:36:36 Distilling Licenses 4.0 deed HTML, legal code HTML, and RDF/XML
244+
INFO 17:36:41 Distilling Licenses 3.0 deed HTML, legal code HTML, and RDF/XML
245+
INFO 17:37:23 Distilling Licenses 2.5 deed HTML, legal code HTML, and RDF/XML
246+
INFO 17:37:50 Distilling Licenses 2.1 deed HTML, legal code HTML, and RDF/XML
247+
INFO 17:37:54 Distilling Licenses 2.0 deed HTML, legal code HTML, and RDF/XML
248+
INFO 17:38:14 Distilling Licenses 1.0 deed HTML, legal code HTML, and RDF/XML
249+
INFO 17:38:20 Distilling Public Domain all deed HTML, legal code HTML, and RDF/XML
250+
INFO 17:38:21 Writing Apache2 redirects configuration
251+
```
252+
- (note that the laptop environment and docker environment had different
253+
timezones, CEST and UTC)
228254

229255

230256
#### Publishing Changes to Git Repo
231257

232-
When the site is deployed, to enable pushing and pulling the licenses data repo
258+
~~When the site is deployed, to enable pushing and pulling the licenses data repo
233259
with GitHub, create an SSH deploy key for the cc-legal-tools-data repo with
234260
write permissions, and put the private key file (not password protected)
235261
somewhere safe (owned by `www-data` if on a server), and readable only by its
236262
owner (0o400). Then in settings, make `TRANSLATION_REPOSITORY_DEPLOY_KEY` be
237-
the full path to that deploy key file.
263+
the full path to that deploy key file.~~ (not active nor current)
238264

239265

240266
#### Publishing Dependency Documentation
@@ -372,24 +398,36 @@ These resources are available for developing this tooling:
372398
### Helper Scripts
373399

374400
Best run before every commit:
375-
- `./dev/coverage.sh` - Run coverage tests and report
376-
- `./dev/tools.sh` - Run Python code tools (isort, black, flake8)
401+
- `./bin/coverage.sh` - Run coverage tests and report
402+
- `./bin/tools.sh` - Run Python code tools (isort, black, flake8)
377403

378404
Run as needed:
379-
- `./dev/copy_theme.sh` - Copy the portions of
405+
- `./bin/copy_theme.sh` - Copy the portions of
380406
[creativecommons/vocabulary-theme][vocab-theme] needed for local development
381407
- Run after each new release of
382408
[creativecommons/vocabulary-theme][vocab-theme]
383409

410+
Generate static files:
411+
- `./bin/publish.sh` - Run Django publish mangement command via Docker
412+
413+
Django manage via Docker:
414+
- `./bin/manage.py` - Run Django management commands via Docker
415+
- Example:
416+
```shell
417+
./bin/manage.py -h
418+
```
419+
384420
Data management:
385-
- `./dev/dump_data.sh` - Dump Django application data
386-
- `./dev/init_data.sh` - :warning: Initialize Django application data
387-
- `./dev/load_data.sh` - Load Django application data
421+
- `./bin/dump_data.sh` - Dump Django application data
422+
- `./bin/init_data.sh` - :warning: Initialize Django application data
423+
- `./bin/load_data.sh` - Load Django application data
424+
425+
388426

389427
Esoteric and dangerous:
390-
- `./dev/updatemessages.sh` - :warning: Run Django Management
391-
nofuzzy_makemessages with helpful options (including excluding legalcode) and
392-
compilemessages
428+
- `./bin/updatemessages.sh` - :warning: Run Django Management
429+
`nofuzzy_makemessages` with helpful options (ex. excluding legalcode) and
430+
`compilemessages`
393431

394432
[vocab-theme]: https://github.com/creativecommons/vocabulary-theme
395433

‎dev/20230901_norm_legacy_rdf.py renamed to ‎bin/20230901_norm_legacy_rdf.py

File renamed without changes.

‎dev/20231009_concatenatemessages.sh renamed to ‎bin/20231009_concatenatemessages.sh

File renamed without changes.

‎dev/20231129_modify_deeds_for_footnotes.sh renamed to ‎bin/20231129_modify_deeds_for_footnotes.sh

File renamed without changes.

‎dev/20250210_update_legacy_cc0.sh renamed to ‎bin/20250210_update_legacy_cc0.sh

File renamed without changes.

‎dev/check_python_versions.sh renamed to ‎bin/check_python_versions.sh

+16-6
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,21 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22
#
33
# Check if all specified Python versions match Pipfile
44
#
5+
#### SETUP ####################################################################
6+
7+
set -o errexit
58
set -o errtrace
69
set -o nounset
710

11+
# shellcheck disable=SC2154
12+
trap '_es=${?};
13+
printf "${0}: line ${LINENO}: \"${BASH_COMMAND}\"";
14+
printf " exited with a status of ${_es}\n";
15+
exit ${_es}' ERR
16+
17+
18+
DIR_REPO="$(cd -P -- "${0%/*}/.." && pwd -P)"
819
# https://en.wikipedia.org/wiki/ANSI_escape_code
920
E0="$(printf "\e[0m")" # reset
1021
E91="$(printf "\e[91m")" # bright red foreground
@@ -13,6 +24,7 @@ E97="$(printf "\e[97m")" # bright white foreground
1324
E100="$(printf "\e[100m")" # bright black (gray) background
1425
EXIT_STATUS=0
1526

27+
#### FUNCTIONS ################################################################
1628

1729
pyvercompare() {
1830
local _path=${1}
@@ -27,7 +39,7 @@ pyvercompare() {
2739
# compare Python version
2840
if [[ "${_path}" == 'Pipfile' ]]
2941
then
30-
_status="✅ ${E92}athoritative Python version${E0}"
42+
_status="✅ ${E92}authoritative Python version${E0}"
3143
PIPFILE_VER=${_ver}
3244
elif [[ "${_ver}" == "${PIPFILE_VER}" ]] \
3345
|| [[ "${_ver}" == "python${PIPFILE_VER}" ]] \
@@ -46,11 +58,9 @@ pyvercompare() {
4658
echo
4759
}
4860

61+
#### MAIN #####################################################################
4962

50-
# Change directory to root directory of cc-legal-tools-app repository
51-
# (grandparent directory of this script)
52-
# shellcheck disable=SC2164
53-
cd "${0%/*}"/../
63+
cd "${DIR_REPO}"
5464

5565
pyvercompare 'Pipfile' 'python_version =' '3'
5666
pyvercompare '.github/workflows/django-app-coverage.yml' 'python-version:' '2'

‎dev/copy_theme.sh renamed to ‎bin/copy_theme.sh

+19-21
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
2+
#
3+
# Copy vocabulary-theme files from a normal checkout (not a releaes tag
4+
# checkout, which has a different structure)
5+
#
6+
#### SETUP ####################################################################
7+
28
set -o errexit
39
set -o errtrace
410
set -o nounset
@@ -9,17 +15,17 @@ trap '_es=${?};
915
printf " exited with a status of ${_es}\n";
1016
exit ${_es}' ERR
1117

18+
DIR_REPO="$(cd -P -- "${0%/*}/.." && pwd -P)"
1219
# https://en.wikipedia.org/wiki/ANSI_escape_code
1320
E0="$(printf "\e[0m")" # reset
14-
E1="$(printf "\e[1m")" # bold
1521
E9="$(printf "\e[9m")" # strike (not supported in Terminal.app)
1622
E30="$(printf "\e[30m")" # black foreground
1723
E31="$(printf "\e[31m")" # red foreground
18-
E33="$(printf "\e[33m")" # yellow foreground
1924
E90="$(printf "\e[90m")" # bright black (gray) foreground
2025
E97="$(printf "\e[97m")" # bright white foreground
2126
E100="$(printf "\e[100m")" # bright black (gray) background
2227
E107="$(printf "\e[107m")" # bright white background
28+
# shellcheck disable=SC2016
2329
README='# Dev theme files
2430
2531
This directory is only used in the Django and GitHub development environments.
@@ -42,9 +48,8 @@ THEME_DIR=''
4248

4349
#### FUNCTIONS ################################################################
4450

45-
4651
copy_vocabulary_theme_files() {
47-
header 'Copying necessary files from vocabulary-themes'
52+
print_header 'Copying necessary files from vocabulary-themes'
4853
print_var REPO_DIR
4954
print_var STATIC_THEME_DIR | sed -e"s#${REPO_DIR}#.#"
5055
{
@@ -56,9 +61,8 @@ copy_vocabulary_theme_files() {
5661
echo
5762
}
5863

59-
6064
create_static_theme_dirs() {
61-
header 'Creating necessary static theme directories'
65+
print_header 'Creating necessary static theme directories'
6266
print_var REPO_DIR
6367
print_var STATIC_DIR | sed -e"s#${REPO_DIR}#.#"
6468
{
@@ -67,26 +71,23 @@ create_static_theme_dirs() {
6771
echo
6872
}
6973

70-
7174
create_wp_content_readme() {
72-
header 'Creating wp-content README.md'
75+
print_header 'Creating wp-content README.md'
7376
print_var REPO_DIR
7477
print_var STATIC_DIR | sed -e"s#${REPO_DIR}#.#"
7578
echo "${README}" > "${STATIC_DIR}/wp-content/README.md"
7679
echo "${E90}STATIC_DIR${E0}/wp-content/README.md"
7780
echo
7881
}
7982

80-
8183
error_exit() {
8284
# Echo error message and exit with error
8385
echo -e "${E31}ERROR:${E0} ${*}" 1>&2
8486
exit 1
8587
}
8688

87-
8889
get_vocabulary_theme_dir() {
89-
header 'Getting vocabulary-theme dir'
90+
print_header 'Getting vocabulary-theme dir'
9091
if ! THEME_DIR="$(cd -P -- \
9192
"${REPO_DIR}"/../vocabulary-theme 2> /dev/null \
9293
&& pwd -P)" || ! [[ -d "${THEME_DIR}" ]]
@@ -100,47 +101,44 @@ get_vocabulary_theme_dir() {
100101
echo
101102
}
102103

103-
104-
header() {
104+
print_header() {
105105
# Print 80 character wide black on white heading with time
106106
printf "${E30}${E107} %-71s$(date '+%T') ${E0}\n" "${@}"
107107
}
108108

109-
110109
print_key_val() {
111110
printf "${E97}${E100}%18s${E0} %s\n" "${1}:" "${2}"
112111
}
113112

114-
115113
print_var() {
116114
print_key_val "${1}" "${!1}"
117115
}
118116

119-
120117
purge_static_theme_dir() {
121-
header 'Purging existing static theme directories'
118+
print_header 'Purging existing static theme directories'
122119
print_var REPO_DIR
123120
print_var STATIC_DIR | sed -e"s#${REPO_DIR}#.#"
124121
print_var STATIC_THEME_DIR | sed -e"s#${REPO_DIR}#.#"
125122
{
126-
rm -f -r -v "${STATIC_THEME_DIR}/"*
123+
rm -f -r -v "${STATIC_THEME_DIR:?}/"*
127124
} | sed \
128125
-e"s#${STATIC_THEME_DIR}#${E90}${E9}STATIC_THEME_DIR${E0}${E9}#" \
129126
-e"s#\$#${E0}#"
130127
print_var REPO_DIR
131128
print_var STATIC_DIR | sed -e"s#${REPO_DIR}#.#"
132129
print_var STATIC_THEME_DIR | sed -e"s#${REPO_DIR}#.#"
133130
{
134-
rm -f -r -v "${STATIC_DIR}/wp-content"
131+
rm -f -r -v "${STATIC_DIR:?}/wp-content"
135132
} | sed \
136133
-e"s#${STATIC_DIR}#${E90}${E9}STATIC_DIR${E0}${E9}#" \
137134
-e"s#\$#${E0}#"
138135
echo
139136
}
140137

141-
142138
#### MAIN #####################################################################
143139

140+
cd "${DIR_REPO}"
141+
144142
get_vocabulary_theme_dir
145143
purge_static_theme_dir
146144
create_static_theme_dirs

‎dev/coverage.sh renamed to ‎bin/coverage.sh

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22
#
33
# Run coverage tests and report
44
#
55
# This script passes all arguments to Coverage Tests. For example, it can be
66
# called from the cc-legal-tools-app directory like so:
77
#
88
# ./dev/coverage.sh --failfast
9+
#
10+
#### SETUP ####################################################################
911

1012
set -o errexit
1113
set -o errtrace
@@ -50,6 +52,7 @@ docker compose exec app coverage erase --debug=dataio
5052
echo
5153

5254
print_header 'Coverage tests'
55+
# shellcheck disable=SC2068
5356
docker compose exec app coverage run --debug=pytest \
5457
manage.py test --noinput --parallel 4 ${@:-} \
5558
|| exit

‎dev/dump_data.sh renamed to ‎bin/dump_data.sh

+10-9
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,9 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22
#
33
# Dump Django application data
44
#
5+
#### SETUP ####################################################################
6+
57
set -o errexit
68
set -o errtrace
79
set -o nounset
@@ -12,6 +14,8 @@ trap '_es=${?};
1214
printf " exited with a status of ${_es}\n";
1315
exit ${_es}' ERR
1416

17+
DATA_FILE='../cc-legal-tools-data/config/app_data.yaml'
18+
DIR_REPO="$(cd -P -- "${0%/*}/.." && pwd -P)"
1519
# https://en.wikipedia.org/wiki/ANSI_escape_code
1620
E0="$(printf "\e[0m")" # reset
1721
E30="$(printf "\e[30m")" # black foreground
@@ -20,7 +24,6 @@ E107="$(printf "\e[107m")" # bright white background
2024

2125
#### FUNCTIONS ################################################################
2226

23-
2427
check_docker() {
2528
local _msg
2629
if ! docker compose exec app true 2>/dev/null; then
@@ -30,29 +33,27 @@ check_docker() {
3033
fi
3134
}
3235

33-
3436
error_exit() {
3537
# Echo error message and exit with error
3638
echo -e "${E31}ERROR:${E0} ${*}" 1>&2
3739
exit 1
3840
}
3941

40-
4142
print_header() {
4243
# Print 80 character wide black on white heading with time
4344
printf "${E30}${E107}# %-70s$(date '+%T') ${E0}\n" "${@}"
4445
}
4546

46-
4747
#### MAIN #####################################################################
4848

49+
cd "${DIR_REPO}"
50+
4951
check_docker
50-
print_header 'Export data (LegalCode and Tool models)'
51-
data_file='../cc-legal-tools-data/config/app_data.yaml'
52+
print_header 'Django dumpdata - Export LegalCode and Tool models'
5253
docker compose exec app ./manage.py dumpdata \
5354
--format yaml \
5455
--indent 2 \
55-
--output "${data_file}" \
56+
--output "${DATA_FILE}" \
5657
legal_tools.LegalCode legal_tools.Tool
57-
du -h "${data_file}"
58+
du -h "${DATA_FILE}"
5859
echo

0 commit comments

Comments
 (0)