Skip to content

Commit f1c0c1d

Browse files
kgodeykgodey
authored
Merge branch 'master' into patch-1
2 parents 068341c + 177229e commit f1c0c1d

File tree

4 files changed

+334
-0
lines changed

4 files changed

+334
-0
lines changed

‎content/cc-search/contents.lr

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
_model: page
2+
---
3+
_template: search_roadmap.html
4+
---
5+
title: CC Search

‎databags/search_roadmap.json

+244
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,244 @@
1+
{
2+
"quarters": [
3+
{
4+
"name": "Q2 2020",
5+
"tasks": [
6+
{
7+
"description": "Plan for potential integration of OER into CC Search through research and consideration of potential issues.",
8+
"gid": "1147157859839985",
9+
"name": "OER Planning"
10+
},
11+
{
12+
"description": "Make changes to the search algorithm that incorporate image popularity data gathered from sources that provide it. ",
13+
"gid": "1149385618454696",
14+
"name": "Improve Search Algorithm with Popularity Data Integration"
15+
},
16+
{
17+
"description": "Plan out search algorithm changes to incorporate image metadata generated via AWS Rekognition.",
18+
"gid": "1154270978154720",
19+
"name": "Plan search algorithm changes for new metadata [AWS Grant]"
20+
},
21+
{
22+
"description": "Improve data processing infrastructure in the Catalog by parallelizing loading and moving storage of data files from providers to S3.",
23+
"gid": "1153114910798065",
24+
"name": "Catalog Infrastructure Improvements"
25+
},
26+
{
27+
"description": "Update Catalog schema to include new metadata generated through AWS Rekognition.",
28+
"gid": "1154270978154717",
29+
"name": "Implement architecture for schema for new metadata [AWS Grant]"
30+
},
31+
{
32+
"description": "Develop metrics for and select a set of ~100 million high quality images for which we'll generate additional metadata through AWS Rekognition.",
33+
"gid": "1154270978154715",
34+
"name": "Image Selection for Rekognition [AWS Grant]"
35+
},
36+
{
37+
"description": "Manage Catalog deployment and provisioning entirely through infrastructure as code.",
38+
"gid": "1167425798148811",
39+
"name": "Improve Catalog Deployment and Provisioning"
40+
},
41+
{
42+
"description": "Switch our Catalog data ingestion for Wikimedia Commons to use the data dumps provided by Wikimedia instead of the MediaWiki API.",
43+
"gid": "1167425798148807",
44+
"name": "Use Data Dumps for Wikimedia Ingestion"
45+
},
46+
{
47+
"description": "Update our Common Crawl provider infrastructure to:\n(1) use Apache Airflow instead of AWS tools like Data Pipeline & Glue for processing data\n(2) unify provider processing to use the same base classes as API providers",
48+
"gid": "1167425798148813",
49+
"name": "Improve Common Crawl Infrastructure"
50+
},
51+
{
52+
"description": "Create better documentation for community contributors by consolidating internal and public documentation and making it available for everyone.",
53+
"gid": "1167425798148815",
54+
"name": "Improve Documentation for Community Contributors"
55+
},
56+
{
57+
"description": "Save popularity data (views, comments, uses, etc.) associated with images from our sources into the Catalog's database.",
58+
"gid": "1164690970919355",
59+
"name": "Machine Processing - Popularity Data to Catalog [AWS Grant]"
60+
},
61+
{
62+
"description": "Move our data cleaning code from the ingestion step of the API to the initial data processing step of the Catalog to eliminate unnecessary repetitive data cleaning.",
63+
"gid": "1167425798148805",
64+
"name": "Move data cleaning pipeline from API to Catalog"
65+
},
66+
{
67+
"description": "We need a frontend feature where users can report problematic content, backend support, and an internal process for taking action on content that is reported as problematic.",
68+
"gid": "1149385618454673",
69+
"name": "User Reporting Strategy & Implementation"
70+
},
71+
{
72+
"description": "Designing and prototyping an upcoming user interface for searching for audio on CC Search.",
73+
"gid": "1163392248010945",
74+
"name": "Design Sprint: Audio UI for CC Search"
75+
},
76+
{
77+
"description": "Design and user test UIs for audio. Ingest a pilot collection of audio to the Catalog, build support in the API. Integrate design to frontend to allow users to search for CC licensed audio.",
78+
"gid": "1171015130050099",
79+
"name": "Audio Support and Integration"
80+
},
81+
{
82+
"description": "Build infrastructure necessary for internationalization, to allow CC Search to be accessible in other languages.",
83+
"gid": "1149456632174198",
84+
"name": "Internationalization Infrastructure"
85+
},
86+
{
87+
"description": "Make accessibility improvements to the UI.",
88+
"gid": "1171784517466065",
89+
"name": "Accessibility Improvements"
90+
},
91+
{
92+
"description": "Create a public version of the CC Search roadmap on CC Open Source.",
93+
"gid": "1168710625773511",
94+
"name": "JSON Export to CC Open Source for Public Roadmap"
95+
},
96+
{
97+
"description": "Integrate License Language Changes into CC Search frontend, which include tooltips on license filters, and adjustments to the language and CTAs on single result pages.",
98+
"gid": "1168725971351190",
99+
"name": "Integration of Design Sprint: License Language Changes"
100+
},
101+
{
102+
"description": "Integrating meta search functionality into CC Search for sources that are not currently indexed, and content types we do not currently support.",
103+
"gid": "1174575887784290",
104+
"name": "Design Sprint: Meta Search Integration"
105+
},
106+
{
107+
"description": "Design and build an embed of CC Search that can be placed on any website, as a starting point to discover objects in CC Search. Components from Design Library must be used, with the goal of simplicity.",
108+
"gid": "1168725971351188",
109+
"name": "CC Search HTML Embed"
110+
},
111+
{
112+
"description": "Offline Old Search (oldsearch.creativecommons.org) and redirect traffic to CC Search. Prior to this, build in messaging on Old Search, and support similar functionality on CC Search. See \"Meta Search Integration\" for related work.",
113+
"gid": "1149456632174214",
114+
"name": "Offline old CC Search"
115+
},
116+
{
117+
"description": "Improve the support pages on CC Search, which includes the Collections page, for a better experience. Add explanation text for collections, improve flow.",
118+
"gid": "1149385618454685",
119+
"name": "Improved Support Pages"
120+
},
121+
{
122+
"description": "Improve how and where we explain licenses, and consider ways to make it easier for reusers to understand and comply with license requirements.",
123+
"gid": "1147666754358269",
124+
"name": "License Explanation/Compliance Improvements"
125+
}
126+
]
127+
},
128+
{
129+
"name": "Q3 2020",
130+
"tasks": [
131+
{
132+
"description": "Research, mock up, and user test potential integrations for Web Monetization into CC Search and other CC web properties.",
133+
"gid": "1153114910798067",
134+
"name": "Web Monetization: Research Phase"
135+
},
136+
{
137+
"description": "Build a UI for the Catalog API, where users can sign up, manage access, see usage metrics and statistics.",
138+
"gid": "1149478266493761",
139+
"name": "API UI with Usage Dashboard"
140+
},
141+
{
142+
"description": "Make CC Catalog API documentation more accessible to CC Search users, and improve user experience.",
143+
"gid": "1164969092703369",
144+
"name": "API documentation improvements"
145+
},
146+
{
147+
"description": "Store a private copy of all the images in the CC Catalog to analyze via machine learning.",
148+
"gid": "1154270978154722",
149+
"name": "Scraping & Resizing Work [AWS Grant]"
150+
},
151+
{
152+
"description": "Collect and use structured data from Wikidata to enhance our search algorithm with semantic search.",
153+
"gid": "1167425798148823",
154+
"name": "Wikidata integration with Catalog & Search Algorithm"
155+
},
156+
{
157+
"description": "Build an analytics UI that is fed by Google Analytics and our internal analytics database.",
158+
"gid": "1149385618454692",
159+
"name": "Usage/Reuse Metrics Dashboard"
160+
},
161+
{
162+
"description": "For all possible providers, use their APIs to ingest data into the CC Catalog instead of scraping websites via Common Crawl data.",
163+
"gid": "1149385618454708",
164+
"name": "Switch from Common Crawl to API"
165+
},
166+
{
167+
"description": "Generate metadata via machine learning (using AWS Rekognition) on a set of ~100 million high quality images from the CC Catalog.",
168+
"gid": "1154270978154727",
169+
"name": "Run Rekognition on 100m images [AWS Grant]"
170+
},
171+
{
172+
"description": "Upgrade the CC Catalog database to use a schema-less database instead of the relational database (Postgres) that we currently use.",
173+
"gid": "1167425798148817",
174+
"name": "Upgrade Catalog: Data Lake"
175+
},
176+
{
177+
"description": "Automate the process of finding new providers of CC-licensed content to index into the CC Catalog.",
178+
"gid": "1167425798148819",
179+
"name": "Provider Review Automation"
180+
},
181+
{
182+
"description": "Implement changes to CC Search (frontend) and Catalog to make use of thumbnails, as they become available.",
183+
"gid": "1154270978154725",
184+
"name": "Implement Use of Thumbnails in Search & Catalog [AWS Grant]"
185+
},
186+
{
187+
"description": "Prepare partnership guidelines for CC Search. Create a page on CC Search publishing these guidelines.",
188+
"gid": "1146971105237802",
189+
"name": "Partnership guidelines for all integration types"
190+
},
191+
{
192+
"description": "Design updates to the CC Search UI in response to new metadata available as a result of applying machine learning to selected images in the Catalog. At a minimum, we expect new filters will be an option. Integration of design will take place subsequently.",
193+
"gid": "1154270978154729",
194+
"name": "Plan UI Updates in Response to Metadata [AWS Grant]"
195+
}
196+
]
197+
},
198+
{
199+
"name": "Q4 2020",
200+
"tasks": [
201+
{
202+
"description": "Do a pilot integration of text-based content that is considered educational. Requires selection of source, Catalog and API structuring, frontend designs and integration.",
203+
"gid": "1144967092198600",
204+
"name": "Text Support and Integration"
205+
},
206+
{
207+
"description": "Plan out the usage of scraping ccREL metadata from the internet to index new content into the CC Catalog.",
208+
"gid": "1164969092703371",
209+
"name": "Plan use of ccREL for easily adding content to cccatalog"
210+
},
211+
{
212+
"description": "Update CC Search user personas based on user research during 2020.",
213+
"gid": "1149385618454694",
214+
"name": "User Persona Redevelopment"
215+
},
216+
{
217+
"description": "Design and implement seamless support for multiple languages in CC Search, as content in languages becomes available. This is preceded by Internationalization Infrastructure work.",
218+
"gid": "1146971105237778",
219+
"name": "Support multiple languages in CC Search\n"
220+
},
221+
{
222+
"description": "Implement design updates to the CC Search UI. Designs will be created in response to new metadata available as a result of applying machine learning to selected images in the Catalog. At a minimum, we expect new filters to be rolled out.",
223+
"gid": "1154270978154731",
224+
"name": "Implement UI Updates for new Metadata [AWS Grant]"
225+
},
226+
{
227+
"description": "Update our search algorithm to use metadata gathered using machine learning analysis (using AWS Rekognition).",
228+
"gid": "1154270978154736",
229+
"name": "Implement Search Algorithm Changes [AWS Grant]"
230+
},
231+
{
232+
"description": "Release the infrastructure code used to power the CC Catalog, API, and CC Search projects publicly.",
233+
"gid": "1154346889867710",
234+
"name": "Ensure Infrastructure Code is Open Source [AWS Grant]"
235+
},
236+
{
237+
"description": "Enrich CC Catalog and data found in the wild using Common Crawl, for example, to track where CC-licensed content is reused.",
238+
"gid": "1167425798148821",
239+
"name": "Enrich CC Catalog data with data from Common Crawl"
240+
}
241+
]
242+
}
243+
]
244+
}

‎templates/layout.html

+3
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,9 @@
102102
{% endfor %}
103103
</div>
104104
</li>
105+
<li class="nav-item{% if this.is_child_of('/cc-search') %} active{% endif%}">
106+
<a class="nav-link" href="{{ '/cc-search'|url }}">CC Search</a>
107+
</li>
105108
<li class="nav-item dropdown {% if this.is_child_of('/archives') %} active{% endif%}">
106109
<a class="nav-link" href="#" id="navbarArchivesDropdown" role="button">Archives</a>
107110
<div class="dropdown-menu" aria-labelledby="navbarArchivesDropdown">

0 commit comments

Comments
 (0)