feed.xml

<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>CC technical blog</title><link href="http://opensource.creativecommons.org/" rel="alternate"></link><link href="http://opensource.creativecommons.org/blog/feed.xml" rel="self"></link><id>urn:uuid:cc6bc3c1-d0ad-365f-b7a6-1fcd73488c56</id><updated>2022-01-06T00:00:00Z</updated><author><name></name></author><entry><title>CC Messaging Update 2022Q1 (Dropping IRC)</title><link href="http://opensource.creativecommons.org/blog/entries/2022-01-06-cc-messaging/" rel="alternate"></link><updated>2022-01-06T00:00:00Z</updated><author><name>TimidRobot</name></author><id>urn:uuid:043a7604-a0bf-3890-b20d-de0f99b67f8c</id><content type="html">&lt;h2 id="past-moved-to-slack"&gt;Past: Moved to Slack&lt;/h2&gt;&lt;p&gt;In 2016, Creative Commons (CC) moved to Slack as our primary messaging platform
(&lt;a href="https://creativecommons.org/2016/10/18/slack-announcement/"&gt;We're on Slack! Join us! - Creative Commons&lt;/a&gt;). We are very thankful
for the generous support that Slack has provided. The Slack messaging platform
is far more accessible than IRC. We saw an immediate and sustained increase in
our messaging community (&lt;a href="https://creativecommons.org/2016/12/09/a-month-of-slack/"&gt;A month of Slack: Growing global communities every
day - Creative Commons,  Lessons learned from a year of Slack, 1000 members,
and immeasurable community growth - Creative Commons&lt;/a&gt;). We
currently have 10,293 members in our Slack workspace. Of those, we see daily
activity from an average of 250 of them spread across almost 70 public
channels. The Slack platform is not without valid criticisms, but those will be
addressed in the Future: Open Source section, below.&lt;/p&gt;
&lt;h2 id="present-dropping-irc"&gt;Present: Dropping IRC&lt;/h2&gt;&lt;p&gt;When CC moved to Slack, we also set up a bridge with our three IRC channels on
Freenode. However those channels only see ones of active users and tens of
messages per year. With the hostile takeover of Freenode in 2021, the
Free/Libre and Open Source (FOSS) community has largely moved to
&lt;a href="https://libera.chat/"&gt;libera.chat&lt;/a&gt;. However, we will not be moving our Slack/IRC bridge
there. &lt;strong&gt;Effective 2022-01-24 we are dropping IRC as an officially supported
messaging platform.&lt;/strong&gt; In addition to there having been very few active users on
IRC, many of the active IRC users also have active Slack accounts. Dropping IRC
will allow us to better allocate our technical resources to better serve the
community as a whole.&lt;/p&gt;
&lt;h2 id="future-open-source"&gt;Future: Open Source&lt;/h2&gt;&lt;p&gt;Over the years, Slack has had performance and UX issues. It is also designed
around assumptions that do not fit a large open community. Those issues have
not prevented it from being a strong and capable messaging platform that has
served our community well. However, an Open Source messaging platform would
better align with the Creative Commons community and the values we champion.
The Open Source and Open Content communities have long enjoyed a significant
overlap and collaboration. With regards to messaging, we hope to increase that
overlap in the next year or two.&lt;/p&gt;
</content></entry><entry><title>Upcoming Changes to the CC Open Source Community</title><link href="http://opensource.creativecommons.org/blog/entries/2020-12-07-upcoming-changes-to-community/" rel="alternate"></link><updated>2020-12-07T00:00:00Z</updated><author><name>kgodey</name></author><id>urn:uuid:f2584ce6-4f24-3ccb-97d8-9f68e62bc65a</id><content type="html">&lt;p&gt;Creative Commons (CC) is adopting a brand new organizational strategy in 2021, just in time for our 20th anniversary. As part of the organization's evolution in alignment with the new strategy, &lt;a href="https://opensource.creativecommons.org/blog/authors/aldenpage/"&gt;Alden Page&lt;/a&gt;, &lt;a href="https://opensource.creativecommons.org/blog/authors/mathemancer/"&gt;Brent Moran&lt;/a&gt;, &lt;a href="https://opensource.creativecommons.org/blog/authors/hugosolar/"&gt;Hugo Solar&lt;/a&gt;, and I (&lt;a href="https://opensource.creativecommons.org/blog/authors/kgodey/"&gt;Kriti Godey&lt;/a&gt;) will have departed Creative Commons by the end of December. Moving forward, the CC staff engineering team of &lt;a href="https://opensource.creativecommons.org/blog/authors/TimidRobot/"&gt;Timid Robot Zehta&lt;/a&gt; and &lt;a href="https://opensource.creativecommons.org/blog/authors/zackkrida/"&gt;Zack Krida&lt;/a&gt; will focus on supporting a smaller set of core projects.&lt;/p&gt;
&lt;p&gt;We are extremely proud of the work we have done together to build CC's vibrant open source community over the past two years. And of course, we're thankful for all the amazing contributions that all our community members have made. We've made significant improvements to existing tools, and launched entirely new projects with your help. &lt;a href="https://opensource.creativecommons.org/blog/categories/cc-vocabulary/"&gt;We created Vocabulary,&lt;/a&gt; a design system for Creative Commons and launched half a dozen sites using it.  We added &lt;a href="https://opensource.creativecommons.org/blog/categories/cc-catalog/"&gt;dozens of new sources to CC Search&lt;/a&gt; and improved &lt;a href="https://opensource.creativecommons.org/blog/authors/AyanChoudhary/"&gt;its accessibility&lt;/a&gt;. We released tools such as the &lt;a href="https://opensource.creativecommons.org/blog/authors/ahmadbilaldev/"&gt;CC WordPress plugin&lt;/a&gt; and &lt;a href="https://opensource.creativecommons.org/blog/authors/makkoncept/"&gt;CC Search browser extension&lt;/a&gt; that integrated CC licensing with widely used software. And, there's so much more.&lt;/p&gt;
&lt;h3 id="community-changes"&gt;Community Changes&lt;/h3&gt;&lt;p&gt;The CC Open Source community remains central to our engineering work, and we will continue to support you in every way we can. However, based on the new staff capacity, we will be making a few changes to our community processes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Community Team members will no longer have access to CC's Asana. Most tasks are tracked on GitHub, and managing Asana adds unnecessary complexity to the community team.&lt;/li&gt;
&lt;li&gt;We will invite all Community Team members to meetings and documents open to the community, regardless of role.&lt;/li&gt;
&lt;li&gt;We will deprecate the  "community-team-core" mailing list in favor of a single "community-team" mailing list.&lt;/li&gt;
&lt;li&gt;We will have a new monthly Open Source Community meeting and cancel the existing biweekly Engineering Meeting.&lt;/li&gt;
&lt;li&gt;We will no longer have a paid Open Source Community Coordinator, &lt;a href="https://opensource.creativecommons.org/community/community-team/community-building-roles/"&gt;relying instead on volunteers&lt;/a&gt; to help assist new community members, maintain our Twitter account, etc.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;We welcome new Community Team members and we will continue to participate in internship programs such as Google Summer of Code.&lt;/p&gt;
&lt;h3 id="project-changes"&gt;Project Changes&lt;/h3&gt;&lt;p&gt;With a smaller engineering team, we will need to support fewer projects. Please see below for the current status of all projects with at least one Community Team member.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Active Development&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;We will continue to actively develop the following projects:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/ccsearch-browser-extension"&gt;CC Search Browser Extension&lt;/a&gt; (maintainer: Mayank Nader)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/creativecommons.github.io-source"&gt;CC Open Source website&lt;/a&gt; (maintainers: Zack Krida &amp;amp; Timid Robot Zehta)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/creativecommons-base"&gt;CC WordPress base&lt;/a&gt; &amp;amp; child themes (new maintainer: Zack Krida)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/legaldb"&gt;CC Legal Database&lt;/a&gt; (maintainer: Timid Robot Zehta)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/chooser"&gt;CC Chooser&lt;/a&gt; (maintainer: Zack Krida)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cc-link-checker/"&gt;CC Link Checker&lt;/a&gt; (maintainer: Timid Robot Zehta)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/licensebuttons/"&gt;License Buttons&lt;/a&gt; (maintainer: Timid Robot Zehta)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/mp/"&gt;Platform Toolkit&lt;/a&gt; (maintainer: Timid Robot Zehta)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/vocabulary"&gt;Vocabulary&lt;/a&gt; (maintainers: Zack Krida &amp;amp; Dhruv Bhanushali)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/wp-plugin-creativecommons"&gt;WordPress Plugin&lt;/a&gt; (new maintainer: Zack Krida)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Maintenance Mode&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;The following projects are entering maintenance mode. The services will remain online, but we will not accept any new pull requests or deploy new code after Dec 15, 2020.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog"&gt;CC Catalog&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-api"&gt;CC Catalog API&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/"&gt;CC Search&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-dataviz/"&gt;Linked Commons&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Catalog, API, and Linked Commons contributors are encouraged to contribute  to our other Python projects such as the &lt;a href="https://github.com/creativecommons/legaldb"&gt;CC Legal Database&lt;/a&gt; or the upcoming &lt;a href="https://github.com/creativecommons/cc-licenses"&gt;CC Licenses&lt;/a&gt; project. If you are a CC Search contributor, we recommend checking out frontend projects such as the &lt;a href="https://github.com/creativecommons/chooser"&gt;CC Chooser&lt;/a&gt; or &lt;a href="https://github.com/creativecommons/vocabulary"&gt;Vocabulary&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id="thank-you"&gt;Thank You!&lt;/h3&gt;&lt;p&gt;We cannot express our gratitude for our community enough. You are all an absolute pleasure to work with, and we're looking forward to continuing to collaborate with you for years to come.&lt;/p&gt;
</content></entry><entry><title>Vocabulary Landing Page &amp; Usage Guide Final Report</title><link href="http://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-closing/" rel="alternate"></link><updated>2020-12-03T00:00:00Z</updated><author><name>nimishbongale</name></author><id>urn:uuid:9c4ce7a3-30fa-397d-bbcd-c2b9ba3658c3</id><content type="html">&lt;p&gt;We have reached the end of this wonderful journey. Let's comprehensively recap all my contributions during the GSoD internship period!&lt;/p&gt;
&lt;h2 id="vocabulary-site-updates-edition-4/4"&gt;Vocabulary Site Updates (Edition 4/4)&lt;/h2&gt;&lt;p&gt;After securing acceptance, I received the necessary github invites. I was given write access to the &lt;a href="https://github.com/creativecommons/vocabulary"&gt;Vocabulary GitHub repository&lt;/a&gt; as a &lt;strong&gt;CC Vocabulary Core Committer&lt;/strong&gt;.&lt;/p&gt;
&lt;h3 id="proposed-initial-plan"&gt;Proposed Initial Plan&lt;/h3&gt;&lt;h4 id="project-synopsis"&gt;Project Synopsis&lt;/h4&gt;&lt;p&gt;Vocabulary has immense potential to be used as a primary UI component library for website building. What it needs is a robust yet layman-friendly how-to guide. Important developer information such as component guides, usage specifications and configuration tweaks form an essential part of any documentation. This will not only encourage existing users to get a feel of how vocabulary continues to grow and reach new milestones, but also promote the usage of Vocabulary in comparatively newer projects. The desired outcomes of my stint as an intern would not only involve penning out a no-nonsense guide to using the pre-existing components but also the designing and developing of a home page (leading to an integrated documentation for each) for Vocabulary, Vue-Vocabulary and Fonts.&lt;/p&gt;
&lt;h3 id="proposed-improvised-timelines-deliverables"&gt;Proposed &amp;amp; Improvised Timelines &amp;amp; Deliverables&lt;/h3&gt;&lt;p&gt;Here's a list of all the weekly goals that I met:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Pre-Internship&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Understood Creative Commons as an organisation, its work and related ethics.&lt;/li&gt;
&lt;li&gt;Had a look at CC’s github repositories and understand the code structure.&lt;/li&gt;
&lt;li&gt;Opened Issues and PR’s to get acquainted with the repository workflows.&lt;/li&gt;
&lt;li&gt;Interacted with my mentor and established the basic ideas regarding the project in question.&lt;/li&gt;
&lt;li&gt;Further researched about the needs of the project, and ponder over its potential impact after implementation.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 1&lt;/strong&gt;
(09/14 - 09/21)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Understood Vocabulary, Vue-Vocabulary and Fonts in greater depth, and their existing components.&lt;/li&gt;
&lt;li&gt;Designed a first look unified landing page for Vocabulary, Vue-Vocabulary and Fonts based on Vocabulary components.&lt;/li&gt;
&lt;li&gt;Interacted with my mentor and other team members and established a rapport.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 2&lt;/strong&gt;
(09/22 - 09/28)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Tackled queries regarding the choice of design, page structure etc., and sought approval from CC’s UX Designer.&lt;/li&gt;
&lt;li&gt;Began to write the content which will need to fill up the main landing page.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 3&lt;/strong&gt;
(09/29 - 10/06)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Finalized the headings, sub-headings and other sections which will need to be present in the landing site &amp;amp; documentation.&lt;/li&gt;
&lt;li&gt;Kept the code ready for accepting documentation contents. Have github pages/netlify/surge configured for continuous integration and deployment.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 4&lt;/strong&gt;
(10/07 - 10/14)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Began to write under “Introduction”, “Getting Started” and ”Grid Components” sub-headings of the documentation.&lt;/li&gt;
&lt;li&gt;Started developing the main landing page using Vocabulary components.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 5&lt;/strong&gt;
(10/15 - 10/22)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Got complete approval for the main page contents.&lt;/li&gt;
&lt;li&gt;Worked on coding the “Dark Theme”.&lt;/li&gt;
&lt;li&gt;Facilitated hacktoberfest contributors and spoke at a CCOS event.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 6&lt;/strong&gt;
(10/23 - 10/30)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Wrote a mid-internship blog post describing work done and how the experience has been so far with CC.&lt;/li&gt;
&lt;li&gt;Started compiling the document guides for all the components in Vocabulary. Made revamps where necessary.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 7&lt;/strong&gt;
(10/31 - 11/07)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Integrated the main page contents and the main landing page itself, had it up and running.  &lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 8&lt;/strong&gt;
(11/08 - 11/15)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Finished writing the Vocabulary usage guide and seek initial approval.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 9&lt;/strong&gt;
(11/16 - 11/23)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Finalized on the guides and the main page contents.&lt;/li&gt;
&lt;li&gt;Carried out the necessary landing page to doc integration.&lt;/li&gt;
&lt;li&gt;Published a sample build using surge for viewing and surveying purposes.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 10&lt;/strong&gt;
(11/24 - 11/30)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Surveyed development builds for Accessibility using WAVE and Accessibility Insights for Web.&lt;/li&gt;
&lt;li&gt;Surveyed the site for responsiveness using Chrome Dev Tools.&lt;/li&gt;
&lt;li&gt;Generated Lighthouse reports.&lt;/li&gt;
&lt;li&gt;Optimised for Search Engines using meta tags and external links.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 11&lt;/strong&gt;
(11/30 - 12/05)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Worked towards improving the report statistics until they reach a respectable target.&lt;/li&gt;
&lt;li&gt;Wrote a blog post summarizing everything, and about my performance cum involvement in CC.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 12&lt;/strong&gt;
(12/06 - 12/12)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Sought daily approvals until everything is finalised.&lt;/li&gt;
&lt;li&gt;Go through my writings and code upteen times for any miniscule errors.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Week 13&lt;/strong&gt;
(12/13 - 12/19)&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Cleaned code, make sure everything is properly linted and ready before the final closing commits.    &lt;/li&gt;
&lt;li&gt;Published the “Concluding Internship” blog post, rounding up my wholesome journey.    &lt;/li&gt;
&lt;li&gt;Sought final closing approval.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Post-Internship&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Promote the use of CC attributed works.&lt;/li&gt;
&lt;li&gt;Interact with the community, answer queries or doubts regarding CC.&lt;/li&gt;
&lt;li&gt;Carry out community work of the repositories I’ve contributed to.&lt;/li&gt;
&lt;li&gt;Leverage experience gained during this internship for future endeavours.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="the-vocabulary-site"&gt;The Vocabulary Site&lt;/h3&gt;&lt;p&gt;Here's the link to &lt;a href="https://cc-vocab-draft.web.app"&gt;the landing site&lt;/a&gt;.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Went through &lt;strong&gt;3&lt;/strong&gt; Design Iterations.&lt;/li&gt;
&lt;li&gt;Designed the mockups in &lt;a href="https://figma.com"&gt;Figma&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Wrote the content filling up the landing page.&lt;/li&gt;
&lt;li&gt;After approval from the UX Designer, waited for an approval from the Frontend Engineer.&lt;/li&gt;
&lt;li&gt;Sought continuous approval from my mentor &lt;a href="https://opensource.creativecommons.org/blog/authors/dhruvkb/"&gt;dhruvkb&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Used &lt;a href="https://vuejs.org"&gt;Vue.js&lt;/a&gt; + &lt;a href="https://www.npmjs.com/package/@creativecommons/vocabulary"&gt;CC Vocabulary&lt;/a&gt; to build a highly modularised site.&lt;/li&gt;
&lt;li&gt;Went through a couple of iterations of the website itself.&lt;/li&gt;
&lt;li&gt;Made about &lt;strong&gt;112&lt;/strong&gt; commits (&lt;strong&gt;15,000&lt;/strong&gt; lines of code) in my &lt;em&gt;gsod-nimish&lt;/em&gt; branch.&lt;/li&gt;
&lt;/ul&gt;
&lt;pre&gt;
&lt;center&gt;
&lt;img alt"Contributions to CC" src="github.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;All my contributions to Creative Commons!&lt;/small&gt;
&lt;/center&gt;
&lt;/pre&gt;&lt;ul&gt;
&lt;li&gt;Used Github API to display repository statistics.&lt;/li&gt;
&lt;/ul&gt;
&lt;pre&gt;
&lt;center&gt;
&lt;img alt"Fetch stats from Github API" src="stats.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;Fetching dynamic stats from the GitHub API&lt;/small&gt;
&lt;/center&gt;
&lt;/pre&gt;&lt;ul&gt;
&lt;li&gt;PR was reviewed and merged on the &lt;strong&gt;25th of November&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here's how the site looks right now:&lt;/p&gt;
&lt;pre&gt;
&lt;center&gt;
&lt;img alt"The final website!" src="website.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;Snapshot of the final website!&lt;/small&gt;
&lt;/center&gt;
&lt;/pre&gt;&lt;ul&gt;
&lt;li&gt;Used &lt;a href="https://surge.sh"&gt;surge&lt;/a&gt; &amp;amp; &lt;a href="https://web.app"&gt;firebase&lt;/a&gt; for draft deploys.&lt;/li&gt;
&lt;li&gt;Carried out &lt;a href="https://developers.google.com/web/tools/lighthouse"&gt;lighthouse&lt;/a&gt; testing.&lt;/li&gt;
&lt;/ul&gt;
&lt;pre&gt;
&lt;center&gt;
&lt;img alt="Lighthouse reports" src="light.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;Lighthouse reports for our live site&lt;/small&gt;
&lt;/center&gt;
&lt;/pre&gt;&lt;ul&gt;
&lt;li&gt;Prompted changes to improve accessibility, SEO and PWA characteristics.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="core-documentation"&gt;Core Documentation&lt;/h3&gt;&lt;p&gt;Here's the link to the &lt;a href="https://cc-vocabulary.netlify.app"&gt;documentation site&lt;/a&gt;.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Used &lt;a href="https://storybook.js.org/"&gt;StorybookJS&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Modified the existing overview page.&lt;/li&gt;
&lt;li&gt;Removed highly verbose sections from the docs. &lt;/li&gt;
&lt;li&gt;Documented Vocabulary sprint planning workflow.&lt;/li&gt;
&lt;li&gt;Documented how to use a markdown component with CC Vocabulary.&lt;/li&gt;
&lt;li&gt;Embedded hyperlink to other open source projects to improve SEO.&lt;/li&gt;
&lt;li&gt;Increased uniformity across documentation present in the storybooks.&lt;/li&gt;
&lt;li&gt;Added alt descriptions &amp;amp; aria labels for certain images to improve accessibility.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="my-learnings-and-challenges"&gt;My Learnings And Challenges&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;Design is more than just picking colors and placing components on a grey screen.&lt;/li&gt;
&lt;li&gt;It's important to read your own writings from an unbiased perspective to actually understand how well it would be perceived.&lt;/li&gt;
&lt;li&gt;Publishing to npmjs is not difficult!&lt;/li&gt;
&lt;li&gt;Knowing the previously existing code in your project is of serious essence. It's important to understand the code styles, structure &amp;amp; activity of the code that you are dealing with.&lt;/li&gt;
&lt;li&gt;Be patient! Its fine to delay something if it makes sense to have it logically accomplished only after certain other tasks are done &amp;amp; dusted with.&lt;/li&gt;
&lt;li&gt;How essential it is to write neat code is something that's not spoken too often. (I wonder why...)&lt;/li&gt;
&lt;li&gt;I always thought Vue.js sets up SPA's by default. I'm surprised you need to configure it additionally to do just that!&lt;/li&gt;
&lt;li&gt;Storybook is just a really nifty OSS with great community support!&lt;/li&gt;
&lt;li&gt;Vue.js is fantastic. Maybe I'm a Vue.js fan now. Should I remain loyal to React? I don't know.&lt;/li&gt;
&lt;li&gt;Making a site responsive isn't the easiest of tasks, but it's certainly doable after a lot of stretching &amp;amp; compressing; lets say that.&lt;/li&gt;
&lt;li&gt;"Code formatting is essential" would be an understatement to make.&lt;/li&gt;
&lt;li&gt;Monorepo's have their own pro's and con's. But in our case the con's were negligible, thankfully!&lt;/li&gt;
&lt;li&gt;GSoD isn't just about documentation; there's some serious amount of coding too!&lt;/li&gt;
&lt;li&gt;You don't have to sit and write code for hours together. Take breaks, come back, and the fix will strike you sooner than ever.&lt;/li&gt;
&lt;li&gt;Timelines change; improvisation being an essential aspect of any project!&lt;/li&gt;
&lt;li&gt;MDX is a neat little format to code in! Documenting code is just so much easier.&lt;/li&gt;
&lt;li&gt;Things become obsolete. Versions become outdated. Code maintaining is therefore, easier said than done!&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="issues-pr-s-raised-during-gsod-period"&gt;Issues &amp;amp; PR's raised during GSoD period&lt;/h3&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Repository&lt;/th&gt; 
&lt;th&gt;Contribution&lt;/th&gt;
&lt;th&gt;Relevant links&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td rowspan=14&gt;&lt;a href="https://github.com/creativecommons/vocabulary"&gt;@creativecommons/vocabulary&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Developed the CC Vocabulary Landing Page&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/pull/747"&gt;https://github.com/creativecommons/vocabulary/pull/747&lt;br&gt;&lt;a href="https://cc-vocab-draft.web.app"&gt;https://cc-vocab-draft.web.app&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Implemented dark mode for our storybooks&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/pull/806"&gt;https://github.com/creativecommons/vocabulary/pull/806&lt;/a&gt;&lt;br&gt;&lt;a href="https://cc-vocabulary.netlify.app"&gt;https://cc-vocabulary.netlify.app&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Carried out a monorepo wide documentation revamp&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/pull/813"&gt;https://github.com/creativecommons/vocabulary/pull/813&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wrote the Monorepo Documentation Story&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/pull/785"&gt;https://github.com/creativecommons/vocabulary/pull/785&lt;/a&gt;&lt;br&gt;&lt;a href="https://cc-vocabulary.netlify.app/?path=/docs/vocabulary-structure--page#why-is-vocabulary-a-monorepo"&gt;https://cc-vocabulary.netlify.app/?path=/docs/vocabulary-structure--page#why-is-vocabulary-a-monorepo&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wrote the Grid Documentation Story&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/pull/802"&gt;https://github.com/creativecommons/vocabulary/pull/802&lt;/a&gt;&lt;br&gt;&lt;a href="https://cc-vocabulary.netlify.app/?path=/docs/layouts-grid--fullhd#grid-system"&gt;https://cc-vocabulary.netlify.app/?path=/docs/layouts-grid--fullhd#grid-system&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wrote the "Getting Started" Usage Guide&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/pull/774"&gt;https://github.com/creativecommons/vocabulary/pull/774&lt;/a&gt;&lt;br&gt;&lt;a href="https://cc-vocabulary.netlify.app/?path=/story/vocabulary-getting-started--page#getting-started"&gt;https://cc-vocabulary.netlify.app/?path=/story/vocabulary-getting-started--page#getting-started&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Added a CHANGELOG.md to adhere to OSS conventions&lt;/td&gt;
&lt;td&gt;&lt;a href=https://github.com/creativecommons/vocabulary/pull/671"&gt;https://github.com/creativecommons/vocabulary/pull/671&lt;/a&gt;&lt;br&gt;&lt;a href="https://github.com/creativecommons/vocabulary/blob/main/CHANGELOG.md"&gt;https://github.com/creativecommons/vocabulary/blob/main/CHANGELOG.md&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unified README.md and updated monorepo build process&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/pull/649"&gt;https://github.com/creativecommons/vocabulary/pull/649&lt;/a&gt;&lt;br&gt;&lt;a href="https://www.npmjs.com/package/@creativecommons/vocabulary"&gt;https://www.npmjs.com/package/@creativecommons/vocabulary&lt;/a&gt;&lt;br&gt;&lt;a href="https://www.npmjs.com/package/@creativecommons/fonts"&gt;https://www.npmjs.com/package/@creativecommons/fonts&lt;/a&gt;&lt;br&gt;&lt;a href="https://www.npmjs.com/package/@creativecommons/vue-vocabulary"&gt;https://www.npmjs.com/package/@creativecommons/vue-vocabulary&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Configured GitHub native dependabot&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/pull/452"&gt;https://github.com/creativecommons/vocabulary/pull/452&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Added phone screen backgrounds&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/pull/445"&gt;https://github.com/creativecommons/vocabulary/pull/445&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Introduce Snapshot Testing to Vocabulary using Chromatic&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/issues/735"&gt;https://github.com/creativecommons/vocabulary/issues/735&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Add a maintained with Lerna badge&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/issues/807"&gt;https://github.com/creativecommons/vocabulary/issues/807&lt;/a&gt;&lt;br&gt;&lt;a href="https://github.com/creativecommons/vocabulary/blob/main/README.md"&gt;https://github.com/creativecommons/vocabulary/blob/main/README.md&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Add new install size badges for our packages&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/issues/776"&gt;https://github.com/creativecommons/vocabulary/issues/776&lt;/a&gt;&lt;br&gt;&lt;a href="https://github.com/creativecommons/vocabulary/blob/main/README.md"&gt;https://github.com/creativecommons/vocabulary/blob/main/README.md&lt;/a&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Customise individual README's for our packages&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/vocabulary/issues/736"&gt;https://github.com/creativecommons/vocabulary/issues/736&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td rowspan=5&gt;&lt;a href="https://github.com/creativecommons/creativecommons.github.io-source"&gt;@creativecommons/creativecommons.github.io-source&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Introductory First Blog Post&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/creativecommons.github.io-source/pull/530"&gt;https://github.com/creativecommons/creativecommons.github.io-source/pull/530&lt;/a&gt;&lt;br&gt;&lt;a href="https://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-intro/"&gt;https://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-intro/&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vocabulary Site Update v1&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/creativecommons.github.io-source/pull/549"&gt;https://github.com/creativecommons/creativecommons.github.io-source/pull/549&lt;/a&gt;&lt;br&gt;&lt;a href="https://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-1/"&gt;https://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-1/&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vocabulary Mid Internship Update v2&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/creativecommons.github.io-source/pull/555"&gt;https://github.com/creativecommons/creativecommons.github.io-source/pull/555&lt;/a&gt;&lt;br&gt;&lt;a href="https://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-2/"&gt;https://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-2/&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vocabulary Site Update v3&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/creativecommons.github.io-source/pull/561"&gt;https://github.com/creativecommons/creativecommons.github.io-source/pull/561&lt;/a&gt;&lt;br&gt;&lt;a href="https://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-3/"&gt;https://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-3/&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vocabulary Site Final Update&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/creativecommons.github.io-source/pull/564"&gt;https://github.com/creativecommons/creativecommons.github.io-source/pull/564&lt;/a&gt;&lt;br&gt;&lt;a href="/"&gt;https://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-closing/&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/cccatalog-api"&gt;@creativecommons/cccatalog-api&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Configured GitHub native dependabot&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/cccatalog-api/pull/53"&gt;https://github.com/creativecommons/cccatalog-api/pull/53&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/ccos-scripts"&gt;@creativecommons/ccos-scripts&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Fix file extension in README.md docs&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/creativecommons/ccos-scripts/pull/100"&gt;https://github.com/creativecommons/ccos-scripts/pull/100&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;p&gt;Follow along my complete GSoD journey through &lt;a href="https://opensource.creativecommons.org/blog/series/gsod-2020-vocabulary-usage-guide/"&gt;these series of posts&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id="memorable-milestones-screenshots"&gt;Memorable Milestones Screenshots&lt;/h3&gt;&lt;pre&gt;
&lt;center&gt;
&lt;img alt"Merged!" src="merged747.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;GSoD PR merged!&lt;/small&gt;
&lt;/center&gt;
&lt;/pre&gt;
&lt;br&gt;
&lt;pre&gt;
&lt;center&gt;
&lt;img alt"Dark Mode" src="darkmode.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;Behold the dark theme!&lt;/small&gt;
&lt;/center&gt;
&lt;/pre&gt;
&lt;br&gt;
&lt;pre&gt;
&lt;center&gt;
&lt;img alt"Grid Docs" src="grid.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;Grid Documenation Story&lt;/small&gt;
&lt;/center&gt;
&lt;/pre&gt;
&lt;br&gt;
&lt;pre&gt;
&lt;center&gt;
&lt;img alt"Monorepo Document Story" src="structure.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;Monorepo Structure Story&lt;/small&gt;
&lt;/center&gt;
&lt;/pre&gt;&lt;h3 id="conclusion"&gt;Conclusion&lt;/h3&gt;&lt;p&gt;My GSoD internship has been by far, a very successful and a fruitful one. I thank the the GSoD team for all their efforts in oragnising it this year. I would also like to thank the entire Creative Commons team for all their motivation and support. The onboarding &amp;amp; see-off was very smooth indeed!&lt;/p&gt;
&lt;p align="center"&gt;
    &lt;strong&gt;Thank you for all your time! This was the final blog post under the Vocabulary docs series. I'll be around for times to come, but until then, sayonara!&lt;/strong&gt;
&lt;/p&gt;</content></entry><entry><title>Summary: My GSoD 2020 Journey</title><link href="http://opensource.creativecommons.org/blog/entries/summary-my-gsod-2020-journey/" rel="alternate"></link><updated>2020-12-02T00:00:00Z</updated><author><name>ariessa</name></author><id>urn:uuid:8ef2254d-edd9-37d9-a144-5b539249b19f</id><content type="html">&lt;p&gt;Thank you for the wonderful experience, Creative Commons!&lt;/p&gt;
&lt;p&gt;This blog post serves as a project report for ‘Improve CC Catalog API Usage Guide’. 
It describes the work that I’ve done during my Google Season Of Docs (GSOD) 2020. 
My mentors for this project are Alden Page and Kriti Godey from Creative Commons.&lt;/p&gt;
&lt;p&gt;In total, there are 12 weeks in the Doc Development Phase. 
Every 2 weeks, I would publish a blog post to update my progress to my mentors and organization.&lt;/p&gt;
&lt;h3 id="week-1"&gt;Week 1&lt;/h3&gt;&lt;p&gt;So, the first two weeks of Google Season of Docs have passed. 
For the first week, I added examples to perform the query using curl command. 
I hit some problem with a Forbidden error. Turns out my access key got expired. 
My problem was solved after obtaining a new access key.&lt;/p&gt;
&lt;h3 id="week-2"&gt;Week 2&lt;/h3&gt;&lt;p&gt;For the second week, I started to write response samples. 
It was tough as I have a hard time understanding drf-yasg, which is an automatic Swagger generator. 
It can produce Swagger / OpenAPI 2.0 specifications from a Django Rest Framework API. 
I tried to find as many examples as I could to increase my understanding. 
Funny, but it took me awhile to realise that drf-yasg is not made up of random letters. 
The DRF part stands for Django Rest Framework while YASG stands for Yet Another Swagger Generator.&lt;/p&gt;
&lt;h3 id="week-3"&gt;Week 3&lt;/h3&gt;&lt;p&gt;Week 3 was quite hectic. 
I moved back to my hometown during week 3. 
Took 3 days off to settle my stuff and set up a workspace. 
I worked on my GSoD project for only 2 days, Monday and Tuesday. 
I managed to create response samples for most API endpoints. 
Had a monthly video call with Kriti this week.&lt;/p&gt;
&lt;h3 id="week-4"&gt;Week 4&lt;/h3&gt;&lt;p&gt;I reviewed what I’ve done and what I haven’t to estimate new completion time. 
Thank god, I have a buffer week in my GSoD timeline and deliverables. 
So yeah, all is good in terms of completion time. 
I started to write descriptions for API endpoints. 
Submitted first PR and published blog entry.&lt;/p&gt;
&lt;h3 id="week-5"&gt;Week 5&lt;/h3&gt;&lt;p&gt;I managed to add a lot of stuff into the documentation. 
I figured out how to add help texts to classes and how to create serializers. 
I also managed to move all code examples under response samples. 
In order to do this, I created a new class called CustomAutoSchema to add x-code-samples. 
Other stuff that I did include creating new sections such as “Register and Authenticate” and “Glossary”. 
The hardest part of this week is probably trying to figure out how to add request body examples and move code examples.&lt;/p&gt;
&lt;h3 id="week-6"&gt;Week 6&lt;/h3&gt;&lt;p&gt;I added another section called Contribute that provides a todolist to start contributing on Github. 
I also wrote and published this blog post.&lt;/p&gt;
&lt;h3 id="week-7"&gt;Week 7&lt;/h3&gt;&lt;p&gt;I restructured the file README in CC Catalog API repository. 
I added a step by step guide on how to run the server locally. 
I hope new users will be less intimidated to contribute to this project with the updated guide on how to run the server locally.&lt;/p&gt;
&lt;h3 id="week-8"&gt;Week 8&lt;/h3&gt;&lt;p&gt;I created Documentation Guidelines which provides steps on how to contribute to CC Catalog API documentation, documentation styles, and cheat sheet for drf-yasg. 
I also wrote and published this blog post.&lt;/p&gt;
&lt;h3 id="week-9"&gt;Week 9&lt;/h3&gt;&lt;p&gt;I had completed all GSoD tasks by week 9. 
So, I took a couple of days off and fixed last week's PR. 
Kriti assigned me with new tasks, which is to port CC Catalog documentation from the internal wiki into GitHub repository. 
Brent, the CC Catalog maintainer explained to me about what needs to be done.&lt;/p&gt;
&lt;h3 id="week-10"&gt;Week 10&lt;/h3&gt;&lt;p&gt;I started exploring CC Catalog and its documentation. 
Reminds me a lot about the first and second weeks of GSoD. 
Trying to understand new stuff and having an "aha" moment when the dots finally connect. 
I started to move the documentation from the internal wiki to CC Catalog’s GitHub repository. 
I also wrote and published this blog post.&lt;/p&gt;
&lt;h3 id="week-11"&gt;Week 11&lt;/h3&gt;&lt;p&gt;I finished working on porting CC Catalog documentation from internal wiki to CC Catalog’s GitHub repository.
Kriti told me that there would be a meeting in which I have to present what I've done for GSoD.
Since the meeting will take place at 1AM in my local time, Kriti told me that I should send a video presentation instead.&lt;/p&gt;
&lt;h3 id="week-12"&gt;Week 12&lt;/h3&gt;&lt;p&gt;I submitted a video presentation to Kriti.
Finished writing project report and evaluation for GSoD.
I published 2 blog posts this week.
One for updates on Week 11 and Week 12.
Another one is this blog post.&lt;/p&gt;
&lt;p&gt;&lt;br/&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;You can view the latest CC Catalog API documentation &lt;a href="https://api.creativecommons.engineering/v1/"&gt;here&lt;/a&gt;.&lt;/p&gt;
</content></entry><entry><title>Finish Video Presentation, Project Report and Evaluation Form</title><link href="http://opensource.creativecommons.org/blog/entries/finish-video-presentation-project-report-and-evaluation-form/" rel="alternate"></link><updated>2020-12-01T00:00:00Z</updated><author><name>ariessa</name></author><id>urn:uuid:3c10027c-ae79-392a-961f-ef9a2362be2a</id><content type="html">&lt;p&gt;For week 10 and 11, I finished porting CC Catalog documentation, submitted a video presentation, and wrapped up my GSoD 2020 journey.&lt;/p&gt;
&lt;h3 id="week-11"&gt;Week 11&lt;/h3&gt;&lt;p&gt;For Week 11, I finished working on porting CC Catalog documentation from internal wiki to CC Catalog’s GitHub repository.
Kriti told me that there would be a meeting in which I have to present what I've done for GSoD.
Since the meeting will take place at 1AM in my local time, Kriti told me that I should send a video presentation instead.&lt;/p&gt;
&lt;h3 id="week-12"&gt;Week 12&lt;/h3&gt;&lt;p&gt;For this week, I submitted a video presentation to Kriti.
Finished writing project report and evaluation for GSoD.
I published 2 blog posts this week.
One for updates on Week 11 and Week 12.
Another one is a summary of my GSoD 2020 journey, which also serves as a project report.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;Signing off.&lt;/p&gt;
</content></entry><entry><title>Presenting CC Base docs - A WordPress Base Theme Usage Guide for the CC Base Theme</title><link href="http://opensource.creativecommons.org/blog/entries/cc-wp-base-theme-docs-launch/" rel="alternate"></link><updated>2020-11-27T00:00:00Z</updated><author><name>JackieBinya</name></author><id>urn:uuid:c27f3a10-a8a1-3cd9-8c66-89cb70d26f58</id><content type="html">&lt;p&gt;We are live 🎉&lt;/p&gt;
&lt;p&gt;The CC Base documentation is live and its available on this &lt;a href="https://cc-wp-theme-base.netlify.app/"&gt;link&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The docs were successfully migrated from Google Docs to the site! One of the most notable changes in the theme and consequently reflected in the documentation is the product name change. The CC WP Theme Base has been renamed to CC Base.&lt;/p&gt;
&lt;p&gt;But the old adage says good documentation is never complete, we hope to engage the Creative Commons Community and perform usability tests. Any feed back gathered from the usability tests will then be used to further improve the CC Base docs.&lt;/p&gt;
&lt;p&gt;In future iterations of the docs development we hope to include the following features:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Increase the quantity of illustrative media so as to make the docs more intuitive this will be marked by adding video tutorials on how to use certain features of the CC Base theme and also adding illustrative tree diagrams to explain hierarchy of key directories and files in the CC Base project structure.&lt;/li&gt;
&lt;li&gt;Integration of &lt;a href="https://www.algolia.com/"&gt;Algolia&lt;/a&gt; a software tool used to power search functionality in static generated sites. &lt;/li&gt;
&lt;li&gt;We also hope to improve SEO for the site.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All the above mentioned improvements are geared at improving the overall user experience of the docs as well as ensure faster onboarding for our community members to get started on using the CC Base theme.&lt;/p&gt;
&lt;p&gt;In conclusion I would like to thank all members of the Creative Commons engineering team, with special mention to Hugo Solar and Kriti Godey. Thank you for your guidance and faith in my abilities as a technical writer and software developer.&lt;/p&gt;
</content></entry><entry><title>Vocabulary Site Updates (Part 3/n)</title><link href="http://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-3/" rel="alternate"></link><updated>2020-11-25T00:00:00Z</updated><author><name>nimishbongale</name></author><id>urn:uuid:9b0d8e22-8041-3b9a-a0dc-66d44cdb924f</id><content type="html">&lt;p&gt;Excited to know more about this week's vocabulary site updates? Read on to find out!&lt;/p&gt;
&lt;h2 id="vocabulary-site-updates-edition-3/many-more-to-come"&gt;Vocabulary Site Updates (Edition 3/many more to come)&lt;/h2&gt;&lt;h3 id="what-i-ve-been-up-to"&gt;What I've been up to&lt;/h3&gt;&lt;center&gt;
&lt;img alt"Halfway There" src="merged.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;The surreal feeling...&lt;/small&gt;
&lt;/center&gt;&lt;p&gt;Merged? Yes. &lt;strong&gt;Merged&lt;/strong&gt;. Here's my story!&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;After getting a thumbs up from the UX Designer, I put up my &lt;a href="https://github.com/creativecommons/vocabulary/pull/747"&gt;GSoD Website PR&lt;/a&gt; for review.   &lt;/li&gt;
&lt;li&gt;I was confident there would be changes, and I let them roll in. It's important to note here that what seems perfect to you may not be so to others, and only experience teaches you the right from the wrong.&lt;/li&gt;
&lt;li&gt;There were a few of them, mainly dealing with spacing, textual content and colors. I resolved them as soon as I could.&lt;/li&gt;
&lt;li&gt;&lt;a href="https://opensource.creativecommons.org/blog/authors/zackkrida/"&gt;zackkrida&lt;/a&gt; has been kind enough to point and enumerate all of them for me!&lt;/li&gt;
&lt;li&gt;After receiving a final approval from the engineering team, my PR was finally merged!&lt;/li&gt;
&lt;li&gt;The final draft of the vocabulary site is live! It will soon be deployed (on &lt;a href="https://netlify,com"&gt;Netlify&lt;/a&gt;) and be made available for public viewing.&lt;/li&gt;
&lt;li&gt;For my readers, here's &lt;a href="https://cc-vocab-draft.web.app"&gt;exclusive preview&lt;/a&gt; of the final draft. &lt;/li&gt;
&lt;li&gt;I've tried making it as optimised as possible, but if you have any inputs whatsoever feel free to raise issues over on our &lt;a href="https://github.com/creativecommons/vocabulary"&gt;GitHub repository&lt;/a&gt;. &lt;/li&gt;
&lt;li&gt;The famed &lt;a href="https://developers.google.com/web/tools/lighthouse"&gt;Lighthouse report&lt;/a&gt; suggests that it's a pretty good start! I've also taken care of the &lt;a href="https://www.w3.org/standards/webdesign/accessibility"&gt;accessibility aspect&lt;/a&gt; wherever applicable.&lt;/li&gt;
&lt;/ul&gt;
&lt;center&gt;
&lt;img alt"Halfway There" src="light.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;Aiming high!&lt;/small&gt;
&lt;/center&gt;&lt;h3 id="what-i-ve-learnt"&gt;What I've learnt&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;GSoD isn't just about documentation; there's some serious amount of coding too!&lt;/li&gt;
&lt;li&gt;You don't have to sit and write code for hours together. Take breaks, come back, and the fix will strike you sooner than ever.&lt;/li&gt;
&lt;li&gt;Timelines change; improvisation being an essential aspect of any project!&lt;/li&gt;
&lt;li&gt;&lt;a href="https://mdxjs.com/"&gt;MDX&lt;/a&gt; is a neat little format to code in! Documenting code is just so much easier.&lt;/li&gt;
&lt;li&gt;Things become obsolete. Versions become outdated. Code maintaining is therefore, easier said than done!&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="other-community-work-tidbits"&gt;Other community work tidbits&lt;/h3&gt;&lt;p&gt;Being a part of an open source organisation also means that I must try to bring in contributions from existing &amp;amp; first time contributors. Here's a peek into my efforts for the same:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The &lt;a href="https://github.com/creativecommons/vocabulary/pull/806"&gt;dark mode PR&lt;/a&gt; started off as a hacktoberfest contribution, and it is now complete!&lt;/li&gt;
&lt;li&gt;Created a &lt;code&gt;/shared&lt;/code&gt; package to house common files between packages (such as the dark &amp;amp; light theme after referring to the &lt;a href="https://reactjs.org/"&gt;React&lt;/a&gt; documentation.&lt;/li&gt;
&lt;li&gt;The automated npm &lt;a href="https://github.com/creativecommons/vocabulary/pull/746"&gt;README.md customisation&lt;/a&gt; is now up and running. (really had a blast solving that issue!) &lt;/li&gt;
&lt;li&gt;If the snapshot testing stands approved, we'll have it running on chromatic!&lt;/li&gt;
&lt;li&gt;Raised issues to add multiple badges to the root README.md  file; namely &lt;code&gt;maintained with Lerna&lt;/code&gt; &amp;amp; custom badges for package sizes from &lt;a href="https://packagephobia.com/"&gt;packagephobia&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p align="center"&gt;
    &lt;strong&gt;Thank you for your time! Stay put for the season finale!&lt;/strong&gt;
&lt;/p&gt;</content></entry><entry><title>Finish GSoD Tasks and Explore CC Catalog Documentation</title><link href="http://opensource.creativecommons.org/blog/entries/finish-gsod-tasks-and-explore-cc-catalog-documentation/" rel="alternate"></link><updated>2020-11-20T00:00:00Z</updated><author><name>ariessa</name></author><id>urn:uuid:d5ea49f0-63f2-3466-93e0-7c513b4dc2d6</id><content type="html">&lt;p&gt;Today marks my fifth blog entry on Creative Commons. 
For week 9 and 10, I explored CC Catalog documentation and began improving the documentation by removing keys and generalizing instructions.&lt;/p&gt;
&lt;h3 id="week-9"&gt;Week 9&lt;/h3&gt;&lt;p&gt;I had completed all GSoD tasks by week 9. 
So, I took a couple of days off and fixed last week's PR. 
Kriti assigned me with new tasks, which is to port CC Catalog documentation from the internal wiki into GitHub repository. 
Brent, the CC Catalog maintainer explained to me about what needs to be done.&lt;/p&gt;
&lt;h3 id="week-10"&gt;Week 10&lt;/h3&gt;&lt;p&gt;For week 10, I started exploring CC Catalog and its documentation. 
Reminds me a lot about the first and second weeks of GSoD. 
Trying to understand new stuff and having an "aha" moment when the dots finally connect. 
I started to move the documentation from the internal wiki to CC Catalog’s GitHub repository. 
I also wrote and published this blog post.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;End of blog entry.&lt;/p&gt;
</content></entry><entry><title>Content Creation Phase: WordPress Base Theme Usage Guide</title><link href="http://opensource.creativecommons.org/blog/entries/cc-wp-base-theme-docs-content-creation/" rel="alternate"></link><updated>2020-11-10T00:00:00Z</updated><author><name>JackieBinya</name></author><id>urn:uuid:bb3168cf-7243-320e-a8e2-0586030dc3a7</id><content type="html">&lt;p&gt;For the past couple of weeks we have been actively creating content for the Creative Commons WordPress Base Theme Usage Guide. Currently the draft content is under final review before it is migrated to the main docs site.&lt;/p&gt;
&lt;h2 id="our-strategy"&gt;Our Strategy&lt;/h2&gt;&lt;p&gt;Our main goal in creating the docs is to create rich, intuitive, engaging, and beautifully presented community facing documentation for the Creative Commons WordPress Base Theme.&lt;/p&gt;
&lt;p&gt;In alignment with the defined goal, our core focus is to create the docs collaboratively.&lt;/p&gt;
&lt;p&gt;The CC WordPress team consists of I, Jacqueline Binya, Hugo Solar and Timid Robot Zehta. Although our team is small it is quite diverse. It consists of a diverse mix of technical skills: I am a junior developer whereas Hugo and Timid are way senior. We also have non-native and native English speakers.&lt;/p&gt;
&lt;p&gt;Diversity is important as we hope to create a high quality product that caters for everyone.&lt;/p&gt;
&lt;p&gt;My role as the tech writer/frontend developer is to create the content: write the documentation, build the docs site and also to create all illustrative media.&lt;/p&gt;
&lt;p&gt;During the content creation phase, the first step involved creating the skeleton of the actual docs site. We created a git branch called &lt;em&gt;docs&lt;/em&gt; within the &lt;a href="https://github.com/creativecommons/wp-theme-base"&gt;creative-commons/wp-base-theme&lt;/a&gt; repository. All content related to the documentation is persisted in that branch. So,please feel free to contribute. We then used &lt;a href="https://gridsome.org/starters/jamdocs/"&gt;JamDocs&lt;/a&gt;, a &lt;a href="https://gridsome.org/"&gt;Gridsome&lt;/a&gt; theme to quickly scaffold the site. We had to adapt the theme so as to make it meet our own specific needs, this involved overhauling the styles and changing the functionality of some of the features in the theme. After that was completed, we then created a &lt;a href="https://docs.google.com/document/d/1yfAQGG70T8BUhZYWglAlQ_lTo4_tYpyjhPN5FsZnSvI/edit?usp=sharing"&gt;Google Doc&lt;/a&gt; we use for collaboratively writing the draft content for the docs site.&lt;/p&gt;
&lt;h2 id="tech-stack"&gt;Tech Stack&lt;/h2&gt;&lt;p&gt;As it was mentioned we used &lt;a href="https://gridsome.org/"&gt;Gridsome&lt;/a&gt; a static generator for &lt;a href="https://vuejs.org/"&gt;Vuejs&lt;/a&gt;. We chose Gridsome because:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;p&gt;We wanted to lower the barrier of entry to contributing:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Gridsome/Vuejs community is very active, help is but a click away.&lt;/li&gt;
&lt;li&gt;The Gridsome official documentation is very resourceful and well maintained.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Gridsome is highly flexible: The content for the actual documentation is written in &lt;a href="https://www.markdownguide.org/getting-started/"&gt;Markdown&lt;/a&gt; but using &lt;a href="https://gridsome.org/plugins/@gridsome/vue-remark"&gt;@gridsome/vue-remark&lt;/a&gt;, which is a Gridsome plugin, we are able to use javascript in Markdown. We intend to include a copy to the clipboard Vuejs component in the site.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Time Constraint: This project is a short running project which has to be completed in a 3 month period. Through the use of JamDocs, a Gridsome templating theme as well various plugins it was easy and fast to get started we were able to add more functionality to the theme with minimal effort.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Ease of integrating &lt;a href="https://cc-vocabulary.netlify.app/"&gt;CC Vocabulary&lt;/a&gt; with Gridsome: it is a requirement that the general aesthetics of all front facing Creative Commons applications is derived from the CC Vocabulary Design System. Major cons for using a design system include the ensuring uniformity in design for all front facing CC products.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="tools-used"&gt;Tools Used&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.figma.com/"&gt;Figma&lt;/a&gt;:&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src="/blog/entries/cc-wp-base-theme-docs-content-creation/image.png" alt="An example of illustrative media"&gt;&lt;/p&gt;
&lt;p&gt;Figma was used to make assets(banners, logos and illustrations) in the theme. The illustrative media was created with accessibility in mind and all the topography used in the illustrative assets was derived from the CC Vocabulary.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://linuxecke.volkoh.de/vokoscreen/vokoscreen.html"&gt;VokoScreenNG&lt;/a&gt;: an open source screencast recording tool used to record all the screen cast demos available in the docs site.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://shotcut.org/"&gt;ShortCut&lt;/a&gt;: an open source video editing tool.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="what-comes-next"&gt;What comes next ?&lt;/h2&gt;&lt;p&gt;After the final review is completed and all feedback implemented we will migrate all the content to the main docs site.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Stay tuned for an update about the launch of the CC WP Base Theme Docs site.&lt;/em&gt;&lt;/p&gt;
</content></entry><entry><title>Vocabulary Site Mid-Internship Update (v2)</title><link href="http://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-2/" rel="alternate"></link><updated>2020-11-09T00:00:00Z</updated><author><name>nimishbongale</name></author><id>urn:uuid:9d7836e4-5dca-3b69-b77e-ba196ec923cc</id><content type="html">&lt;p&gt;This is a mid-internship blog post. Wait. what!? Already? Let's glance over my progress, shall we?&lt;/p&gt;
&lt;h2 id="vocabulary-site-updates-edition-2/many-more-to-come"&gt;Vocabulary Site Updates (Edition 2/many more to come)&lt;/h2&gt;&lt;p&gt;Oh boy! 1.5 months have passed since I've been investing time in building a landing site &amp;amp; usage guide for CC Vocabulary. A lot has changed since the time of posting my last blog post. &lt;strong&gt;A lot&lt;/strong&gt;.&lt;/p&gt;
&lt;center&gt;
&lt;img alt"Halfway There" src="speed.gif"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;Hitting "the point of no return" has never been this exciting! Time to step on the throttle! Source: &lt;a href="https://cliply.co"&gt;Cliply&lt;/a&gt;&lt;/small&gt;
&lt;/center&gt;&lt;h3 id="what-i-ve-been-up-to"&gt;What I've been up to&lt;/h3&gt;&lt;blockquote&gt;&lt;p&gt;&lt;strong&gt;Designing&lt;/strong&gt; ⇨ &lt;strong&gt;Drafting&lt;/strong&gt; ⇨ &lt;strong&gt;Developing&lt;/strong&gt; ⇨ &lt;strong&gt;Debugging&lt;/strong&gt; ⇨ &lt;strong&gt;Deploying&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;And the cycle contrinues. I guess it sums it all up very nicely. &lt;em&gt;Can somebody appreciate the alliteration though?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Here's a gist of what I've achieved so far:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;I've gone through &lt;strong&gt;2&lt;/strong&gt; iterations of the design. I'm happy with how the new site looks (and I genuinely hope the design team does too!).&lt;/li&gt;
&lt;li&gt;I've drafted around &lt;strong&gt;5+&lt;/strong&gt; writeups dealing with Monorepo Migration, Getting Started guide, Vocabulary Overview and of course these blog posts.&lt;/li&gt;
&lt;li&gt;My branch on the vocabulary repository now has over &lt;strong&gt;50+&lt;/strong&gt; commits &amp;amp; over &lt;strong&gt;13,000&lt;/strong&gt; lines of code (not that I've written all of them, but you know, just for the stats)&lt;/li&gt;
&lt;li&gt;The first draft of the vocabulary site is now live! I'm expecting a whole bunch of changes still, but here it is if you want to have a sneak peek: &lt;a href="https://cc-vocab-draft.surge.sh"&gt;https://cc-vocab-draft.surge.sh&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;I've consumed the &lt;a href="https://docs.github.com/en/free-pro-team@latest/rest"&gt;Github API&lt;/a&gt; to get live release history, forks and starrers count. I think it adds a really nice touch to the site in general.&lt;/li&gt;
&lt;li&gt;I've used &lt;a href="https://surge.sh"&gt;surge.sh&lt;/a&gt; to deploy the draft site. I believe it's a really simple tool to have your site deployed within seconds!&lt;/li&gt;
&lt;/ul&gt;
&lt;center&gt;
&lt;img alt"Github commit gif" src="github.png"/&gt;&lt;br&gt;
&lt;small class="muted"&gt;My github contribution chart is filling up!&lt;/small&gt;
&lt;/center&gt;&lt;h3 id="what-i-ve-learnt"&gt;What I've learnt&lt;/h3&gt;&lt;p&gt;Some say it's hard to learn through virtual internships. Well, let me prove you wrong. Here are my leanings in the past few weeks:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;It's surprising how subjective (&amp;amp; yet objective) designing really is.&lt;/li&gt;
&lt;li&gt;Vue.js is &lt;em&gt;fantastic&lt;/em&gt;. Maybe I'm a Vue.js fan now. Should I remain loyal to React? I don't know.&lt;/li&gt;
&lt;li&gt;Making a site responsive isn't the &lt;em&gt;easiest&lt;/em&gt; of tasks, but it's certainly doable after a lot of stretching &amp;amp; compressing; lets say that.&lt;/li&gt;
&lt;li&gt;"Code formatting is essential" would be an &lt;em&gt;understatement&lt;/em&gt; to make.&lt;/li&gt;
&lt;li&gt;Monorepo's have their own pro's and con's. But in our case the con's were negligible, thankfully! &lt;/li&gt;
&lt;li&gt;I'll be following up with some performance &amp;amp; accessibility testing this coming week, so let's see how that plays out!&lt;/li&gt;
&lt;li&gt;A mentor plays a vital role in any project. My mentor &lt;code&gt;@dhruvkb&lt;/code&gt; has been very supportive and has made sure I stick to my timeline!&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="other-community-work-tidbits"&gt;Other community work tidbits&lt;/h3&gt;&lt;p&gt;I believe apart from the internship work that I'm engaged in, I should also help around with some community PR work. I've been told I'm always welcome to, which is great!&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;I got the opportunity to speak at a CCOS event alongwith fellow speakers &lt;a href="https://opensource.creativecommons.org/blog/authors/dhruvkb/"&gt;dhruvkb&lt;/a&gt; &amp;amp; &lt;a href="https://opensource.creativecommons.org/blog/authors/dhruvi16/"&gt;dhruvi16&lt;/a&gt;. I had a blast talking to budding students from DSC-IIT Surat &amp;amp; DSC-RIT.&lt;/li&gt;
&lt;li&gt;The dark mode (as promised) should be out before my next blog post.&lt;/li&gt;
&lt;li&gt;Deployed the vocabulary storybook on &lt;a href="https://chromatic.com"&gt;Chromatic&lt;/a&gt; and compared &amp;amp; contrasted the pros &amp;amp; cons. Snapshot testing in the near future maybe? &lt;/li&gt;
&lt;li&gt;Completed the hacktoberfest challenge. &lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="bonus-content"&gt;Bonus content&lt;/h3&gt;&lt;p&gt;Not many of you may know this, but this site uses the &lt;a href="https://getlektor.com"&gt;Lektor&lt;/a&gt; CMS. I needed to have it installed on my system (windows 10 OS) to run the code in our site repository. 
Lektor suggests running the following code in powershell as an installation step:&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;new-object&lt;/span&gt; &lt;span class="n"&gt;net&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;webclient&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;DownloadString&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;https://www.getlektor.com/installer.py&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt; &lt;span class="n"&gt;python&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;I just didn't think this is a very elegant way. Being an ardent &lt;a href="/blog/entries/cc-vocabulary-docs-updates-2/chocolatey.org"&gt;chocolatey.org&lt;/a&gt; fan, I just had to have it up on there! Now the installation step for lektor is simply:&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;choco&lt;/span&gt; &lt;span class="n"&gt;install&lt;/span&gt; &lt;span class="n"&gt;lektor&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;on the Windows PowerShell!&lt;/p&gt;
&lt;p&gt;Have a look at the package here:&lt;/p&gt;
&lt;p&gt;&lt;a href="https://chocolatey.org/packages/lektor"&gt;https://chocolatey.org/packages/lektor&lt;/a&gt;&lt;/p&gt;
&lt;p align="center"&gt;
    &lt;strong&gt;Thank you for your time! Stay put for the next Vocabulary site update!&lt;/strong&gt;
&lt;/p&gt;</content></entry><entry><title>Restructure README and Add Documentation Guidelines</title><link href="http://opensource.creativecommons.org/blog/entries/restructure-readme-and-add-documentation-guidelines/" rel="alternate"></link><updated>2020-11-05T00:00:00Z</updated><author><name>ariessa</name></author><id>urn:uuid:98db0051-b46f-3e4b-ae18-a56418565d50</id><content type="html">&lt;p&gt;This is my fourth blog entry on Creative Commons. 
For week 7 and 8, I restructured the file README to be more digestible to new users and created Documentation Guidelines for CC Catalog API documentation.&lt;/p&gt;
&lt;h3 id="week-7"&gt;Week 7&lt;/h3&gt;&lt;p&gt;For this week, I restructured the file README in CC Catalog API repository. 
I added a step by step guide on how to run the server locally. 
I hope new users will be less intimidated to contribute to this project with the updated guide on how to run the server locally.&lt;/p&gt;
&lt;h3 id="week-8"&gt;Week 8&lt;/h3&gt;&lt;p&gt;For week 8, I created Documentation Guidelines which provides steps on how to contribute to CC Catalog API documentation, documentation styles, and cheat sheet for drf-yasg. 
I also wrote and published this blog post.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;Finis.&lt;/p&gt;
</content></entry><entry><title>Vocabulary Site Updates (v1)</title><link href="http://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-updates-1/" rel="alternate"></link><updated>2020-10-26T00:00:00Z</updated><author><name>nimishbongale</name></author><id>urn:uuid:51a44307-9725-3d59-bfcd-9badc7cce229</id><content type="html">&lt;p&gt;Hello there! Well well well. It has been an eventful first few weeks, to say the least! Let's gauge my progress, shall we?&lt;/p&gt;
&lt;h2 id="vocabulary-site-updates-edition-1/many-more-to-come"&gt;Vocabulary Site Updates (Edition 1/many more to come)&lt;/h2&gt;&lt;h3 id="what-i-ve-been-upto"&gt;What I've been upto&lt;/h3&gt;&lt;p&gt;I've mainly got myself invested in a survey of the existing documentation that vocabulary currently possesses, and find places where it could be made better. After clearing those issues out, I began building the main landing site for &lt;code&gt;Vocabulary&lt;/code&gt;, &lt;code&gt;Vue-vocabulary&lt;/code&gt; and &lt;code&gt;Fonts&lt;/code&gt;. It wasn't particularly difficult to establish the necessary workflows as I had done something similar before. During the process of designing the basic structure of the site, I came across a few instances where I felt we needed new/improved components &amp;amp; I discussed the same with my team over on the sprint calls. The design of the site is nearly done. I'm also building the site parallelly &amp;amp; seeking approval from the CC Design Team. I've gotten myself involved in multiple other community contributions to CC as well across multiple of our repositories.&lt;/p&gt;
&lt;h3 id="what-i-ve-learnt"&gt;What I've learnt&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;Knowing the previously existing code in your project is of serious essence. It's important to understand the code styles, structure &amp;amp; activity of the code that you are dealing with.&lt;/li&gt;
&lt;li&gt;Be patient! Its fine to delay something if it makes sense to have it logically accomplished only after certain other tasks are done &amp;amp; dusted with.&lt;/li&gt;
&lt;li&gt;How essential it is to write &lt;em&gt;neat code&lt;/em&gt; is something that's not spoken too often. (I wonder why...)&lt;/li&gt;
&lt;li&gt;I always thought VueJS sets up SPA's by default. I'm surprised you need to configure it additionally to do just that!&lt;/li&gt;
&lt;li&gt;Storybook is just a really nifty OSS with great community support! &lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="other-community-work-tidbits"&gt;Other community work tidbits&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;I've been working on the Dark Mode (a much awaited feature, at least for me!) for our storybooks with some support from our community. It should be up and running shortly!&lt;/li&gt;
&lt;li&gt;Fixed some formatting bugs in the &lt;code&gt;README.md&lt;/code&gt; &amp;amp; suggested changes wrt to &lt;code&gt;npm v7&lt;/code&gt; considerations.&lt;/li&gt;
&lt;li&gt;Fixed storybook components docs for 2 features.&lt;/li&gt;
&lt;li&gt;Raised a ticket for a component to render markdown text within vocabulary itself.&lt;/li&gt;
&lt;li&gt;Raised a few other issues for potential hacktoberfest contributions.&lt;/li&gt;
&lt;/ul&gt;
&lt;p align="center"&gt;
    &lt;strong&gt;Thank you for your time! To be continued...&lt;/strong&gt;
&lt;/p&gt;</content></entry><entry><title>Add New Sections, Descriptions, Help Texts, Code Examples, Schemas, and Serializers</title><link href="http://opensource.creativecommons.org/blog/entries/add-new-sections-descriptions-help-texts-code-examples-schemas-and-serializers/" rel="alternate"></link><updated>2020-10-21T00:00:00Z</updated><author><name>ariessa</name></author><id>urn:uuid:b832956d-0c04-37e5-bde6-eb53722a4ba3</id><content type="html">&lt;p&gt;Welcome to my third blog entry! For week 5 and 6, I added new sections, descriptions, help texts, code examples, schemas, and serializers. I was so productive these past two weeks.&lt;/p&gt;
&lt;h3 id="week-5"&gt;Week 5&lt;/h3&gt;&lt;p&gt;For this week, I managed to add a lot of stuff into the documentation. 
I figured out how to add help texts to classes and how to create serializers. 
I also managed to move all code examples under response samples. 
In order to do this, I created a new class called CustomAutoSchema to add &lt;a href="https://github.com/Redocly/redoc/blob/master/docs/redoc-vendor-extensions.md#x-codesamples"&gt;x-code-samples&lt;/a&gt;. 
Other stuff that I did include creating new sections such as “Register and Authenticate” and “Glossary”. 
The hardest part of this week is probably trying to figure out how to add request body examples and move code examples.&lt;/p&gt;
&lt;h3 id="week-6"&gt;Week 6&lt;/h3&gt;&lt;p&gt;For week 6, I added another section called Contribute that provides a todolist to start contributing on Github. 
I also wrote and published this blog post.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;All caught up!&lt;/p&gt;
</content></entry><entry><title>Add Response Samples and Descriptions for API Endpoints</title><link href="http://opensource.creativecommons.org/blog/entries/add-response-samples-and-descriptions-for-api-endpoints/" rel="alternate"></link><updated>2020-10-09T00:00:00Z</updated><author><name>ariessa</name></author><id>urn:uuid:23d6eb6e-e157-34fb-af99-ddfcad7c5044</id><content type="html">&lt;p&gt;Well, hello again 👋! For week 3 and week 4, I added response samples and descriptions for API endpoints. Writing documentation feels a bit like coding at this point because I need to read a lot about drf-yasg, dig through issues and questions at Github / Stackoverflow to ensure that I don’t ask redundant (or even stupid) questions.&lt;/p&gt;
&lt;h3 id="week-3"&gt;Week 3&lt;/h3&gt;&lt;p&gt;Week 3 was quite hectic. I moved back to my hometown during week 3. 
Took 3 days off to settle my stuff and set up a workspace. 
I worked on my GSoD project for only 2 days, Monday and Tuesday. 
I managed to create response samples for most API endpoints. 
Had a monthly video call with Kriti this week.&lt;/p&gt;
&lt;h3 id="week-4"&gt;Week 4&lt;/h3&gt;&lt;p&gt;For this week, I reviewed what I’ve done and what I haven’t to estimate new completion time. 
Thank god, I have a buffer week in my GSoD timeline and deliverables. 
So yeah, all is good in terms of completion time. 
I started to write descriptions for API endpoints.
Submitted first PR and published blog entry.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;Over and out.&lt;/p&gt;
</content></entry><entry><title>Vocabulary Site &amp; Usage Guide Introduction (GSoD'20)</title><link href="http://opensource.creativecommons.org/blog/entries/cc-vocabulary-docs-intro/" rel="alternate"></link><updated>2020-10-02T00:00:00Z</updated><author><name>nimishbongale</name></author><id>urn:uuid:4974b78b-a4f9-3bbc-8875-d595042b954c</id><content type="html">&lt;p&gt;Hey there! I'm Nimish Bongale, a Technical Writer &amp;amp; Software Developer based out of Bangalore, India. My other hobbies include playing chess and the guitar. I look forward to build the CC Vocabulary site and usage guides as a part of GSoD'20.&lt;/p&gt;
&lt;h2 id="but-what-is-gsod"&gt;But what is GSoD?&lt;/h2&gt;&lt;p&gt;GSoD, or Google Season of Docs, is a program that stresses on the importance of the documentation aspect of Open Source projects. It invites technical writers from across the world to submit proposals based on projects floated in by the participating Open Source Organisations. The selected technical writers then work with the their respective organisations and look to complete their work by the end of their internship period. More information about the same can be found &lt;a href="https://developers.google.com/season-of-docs"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Let's talk a bit about my project, shall we?&lt;/p&gt;
&lt;h2 id="vocabulary-site-usage-guide"&gt;Vocabulary Site &amp;amp; Usage Guide&lt;/h2&gt;&lt;h3 id="introduction"&gt;Introduction&lt;/h3&gt;&lt;p&gt;&lt;a href="https://github.com/creativecommons/vocabulary"&gt;CC Vocabulary&lt;/a&gt; is a cohesive design system &amp;amp; Vue component library to unify the web-facing Creative Commons. It's currently comprised of 3 packages, namely Vocabulary, Vue-Vocabulary &amp;amp; Fonts. My contribution to this project would majorly involve building the landing site for CC Vocabulary, and refactor the documentation wherever necessary.&lt;/p&gt;
&lt;h3 id="what-drives-me"&gt;What drives me&lt;/h3&gt;&lt;p&gt;Documentation is one of the primary reasons which determines how successful a certain open source library will be. The major question that developers think of while choosing a suitable tech stack to build their applications is:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Is the library &lt;em&gt;well documented&lt;/em&gt;? &lt;/li&gt;
&lt;li&gt;Is it &lt;em&gt;well maintained&lt;/em&gt;? &lt;/li&gt;
&lt;li&gt;Does it have some &lt;em&gt;considerable usage and error support&lt;/em&gt;?&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These are exactly the questions I should be asking myself while going about this project idea.&lt;/p&gt;
&lt;p&gt;As aforementioned, there is an immanent need to have a concise and consolidated documentation. The lack of documentation hurts the future perspectives of open source applications, and is by far, an essential and non-negligible component. Linking to these documentations should be an appealing home page, which captures the interest of the people in an instant. The documentation should be well organised, thereby enabling a seamless flow through it.&lt;/p&gt;
&lt;h3 id="tech-stack-of-the-project"&gt;Tech stack of the project&lt;/h3&gt;&lt;p&gt;We have decided to move forward with &lt;a href="https://vuejs.org/"&gt;Vuejs&lt;/a&gt; for building the site, and continue work on the existing &lt;a href="https://storybook.js.org/"&gt;storybooks&lt;/a&gt; of Vocabulary, Vue-Vocabulary and Fonts. Storybookjs has had some great improvements in recent times, and the new addons that are offered will greatly support my work. Besides these, I will also be using &lt;a href="https://stackedit.io/"&gt;StackEdit&lt;/a&gt; to write and share Markdown files of my writings.&lt;/p&gt;
&lt;h3 id="progress-baby-steps"&gt;Progress - Baby Steps&lt;/h3&gt;&lt;p&gt;I have contributed to CC in the past. It would now be my first time contributing to a specific project within CC, while being a member of CC Open Source. Some tasks that I've been able to initiate/accomplish so far:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Look at Open Source documentation conventions, and see if we violate any.&lt;/li&gt;
&lt;li&gt;Understand the level of existing documentation currently present in our storybooks.&lt;/li&gt;
&lt;li&gt;Discuss about the Monorepo migration and help out with the implementation.&lt;/li&gt;
&lt;li&gt;Migrate &lt;code&gt;storybookjs&lt;/code&gt; to the latest version.&lt;/li&gt;
&lt;li&gt;Implement &lt;code&gt;addon-controls&lt;/code&gt; for vocabulary.&lt;/li&gt;
&lt;li&gt;Design the vocabulary site.&lt;/li&gt;
&lt;li&gt;Promote the involvement of CC Open Source in &lt;a href="https://hacktoberfest.digitalocean.com/"&gt;Hacktoberfest&lt;/a&gt; 2020.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="what-did-i-learn"&gt;What did I learn?&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;Design is more than just picking colors and placing components on a grey screen.&lt;/li&gt;
&lt;li&gt;It's important to read your own writings from an unbiased perspective to actually understand how well it would be perceived.&lt;/li&gt;
&lt;li&gt;Interacting with your mentor on a regular basis is of the absolute essence.&lt;/li&gt;
&lt;li&gt;Publishing to &lt;a href="/blog/entries/cc-vocabulary-docs-intro/npmjs.com"&gt;npmjs&lt;/a&gt; is not difficult!&lt;/li&gt;
&lt;/ul&gt;
&lt;p align="center"&gt;
    &lt;strong&gt;Thank you for your time!&lt;/strong&gt;
&lt;/p&gt;</content></entry><entry><title>Creative Commons WordPress plugin: attribution for images</title><link href="http://opensource.creativecommons.org/blog/entries/cc-wp-plugin-attribution-for-images/" rel="alternate"></link><updated>2020-10-01T00:00:00Z</updated><author><name>rczajka</name></author><id>urn:uuid:818016d3-7344-3937-9fd6-7e5ffad98071</id><content type="html">&lt;p&gt;As a part of &lt;a href="https://centrumcyfrowe.pl"&gt;Centrum Cyfrowe&lt;/a&gt;'s &lt;a href="https://otwartakultura.org/noworries/"&gt;#NoWorries project&lt;/a&gt; funded by EUIPO, 
I have had the pleasure of enhancing the Creative Commons Wordpress plugin. 
The new version of CC's Wordpress plugin has a feature called
“attribution information for images”. It works like this:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;you upload an image to the Wordpress Media Library and fill out the
correct attribution information there.&lt;/li&gt;
&lt;li&gt;You then insert the image into a page using the Image Gutenberg block.&lt;/li&gt;
&lt;li&gt;When the image is then displayed on site, the plugin will show the
attribution information – the name of the author, the image's title
and link to source, and the CC license used – right there, in a nice
semi-transparent overlay over the image.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="how-does-it-work"&gt;How does it work?&lt;/h2&gt;&lt;p&gt;To find the relevant information from the Media Library, the plugin
reuses the information already provided by Gutenberg Image Blocks.
Each time an image is inserted using such a block, Wordpress adds a
special CSS class to it, in the form of &lt;code&gt;wp-image-{id}&lt;/code&gt;, containing
the image's identifier in the Media Library. It can be used to add
individual styles to a specific image – we're using it to find the
relevant entry in the Media Library and add individual attribution
information. With this approach, we avoid the need for any custom
markup – while also only hitting the database with a query when an
actual image from the Media Library is found on the page.&lt;/p&gt;
&lt;p&gt;All you need to do is make sure the licensing information is there in
the Media Library, and that the images are inserted using the Image
block.&lt;/p&gt;
&lt;p&gt;This wasn't the first attempt at adding a similar function to the CC
Wordpress plugin. The previous attempt used a &lt;code&gt;[license]&lt;/code&gt; shortcode
wrapping the image – which it's unwieldy with the current Wordpress
Gutenberg editor. It also used multiple calls to
&lt;code&gt;attachment_url_to_postid&lt;/code&gt; to locate the image in the Media Library, which
meant executing more database queries for each image. With the new
approach, the user doesn't have to change their posts at all – all
they need to do is install the plugin and add attribution information
in the Media Library, and it will automatically start working for
their normally inserted images.&lt;/p&gt;
&lt;p&gt;See here how to install the plugin:&lt;/p&gt;
&lt;video src="install.mp4" controls&gt;&lt;/video&gt;&lt;p&gt;See here how to use the image attribution function:&lt;/p&gt;
&lt;video src="use.mp4" controls&gt;&lt;/video&gt;</content></entry><entry><title>WordPress Base Theme Usage Guide (GSOD-2020): Hello World!</title><link href="http://opensource.creativecommons.org/blog/entries/cc-wp-base-theme-docs-intro/" rel="alternate"></link><updated>2020-09-30T00:00:00Z</updated><author><name>JackieBinya</name></author><id>urn:uuid:8ca33f24-33d1-3bdb-a55d-5fb6381316f5</id><content type="html">&lt;p&gt;My name is Jacqueline Binya. I am a software developer and technical writer from Zimbabwe. I am going to write a series of blog posts documenting my experience and lessons as I contribute to the &lt;a href="https://github.com/creativecommons/wp-theme-base"&gt;Creative Commons WordPress Base Theme(CC WP Base Theme)&lt;/a&gt; during the &lt;a href="https://developers.google.com/season-of-docs"&gt;Google Season of Docs (GSOD-2020)&lt;/a&gt; as a technical writer.&lt;/p&gt;
&lt;h2 id="what-is-google-season-of-docs"&gt;What is Google Season of Docs?&lt;/h2&gt;&lt;p&gt;The Google Season of the Docs was born out of a need to improve the quality of open-source documentation as well as to advocate for open source, for documentation, and for technical writing. Annually during the GSOD, technical writers are invited to contribute to open-source projects through a highly intensive process geared at ensuring that the technical writers and the projects they contribute to during GSOD are a good fit, after that has been determined GSOD then resumes.&lt;/p&gt;
&lt;h2 id="building-the-docs"&gt;Building the docs&lt;/h2&gt;&lt;p&gt;The CC WP Base theme is a WordPress theme used to create front-facing Creative Commons (CC) websites. My task is to collaborate with the engineering team to create community facing docs for the theme.&lt;/p&gt;
&lt;h3 id="guiding-principles"&gt;Guiding principles&lt;/h3&gt;&lt;p&gt;The docs should be inclusive meaning: they should be written in an easy-to-understand manner taking care to avoid the use of excessive technical jargon, they should be accessible and they should have support for internationalization. We hope to provide our users with a smooth and memorable experience whilst using the docs hence the docs site should be fast and easy to navigate.&lt;/p&gt;
&lt;h3 id="technical-stack-of-the-project"&gt;Technical stack of the project&lt;/h3&gt;&lt;p&gt;We decided to build the docs using &lt;a href="https://jamstack.org/"&gt;Jamstack&lt;/a&gt;, to be specific we are using  &lt;a href="https://gridsome.org/"&gt;Gridsome&lt;/a&gt; a static generator for &lt;a href="https://vuejs.org/"&gt;Vuejs&lt;/a&gt;. We are using Gridsome as it is highly performant, and it also integrates smoothly with the &lt;a href="https://cc-vocabulary.netlify.app/"&gt;CC Vocabulary&lt;/a&gt;. Gridsome also has out-of-the-box support for important features like Google Analytics and &lt;a href="https://www.algolia.com/"&gt;Angolia&lt;/a&gt;, these features will obviously be useful in future iterations of the docs. To quickly scaffold the docs we used a Gridsome theme called &lt;a href="https://gridsome.org/starters/jamdocs/"&gt;JamDocs&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id="progress"&gt;Progress&lt;/h3&gt;&lt;p&gt;Currently, the project is on track. As it's been stated we are creating the docs collaboratively. The very first step in our workflow is to create draft content using Google docs. That task is assigned to me, it involves doing lots of research, reading and also testing out the theme. Afterwards, my mentors Hugo Solar and Timid Robot Zehta then give me feedback on the draft. Then I implement the feedback and continuously work on improvements. The final step is migrating the approved draft content to the docs projects in markdown format.&lt;/p&gt;
&lt;h3 id="my-lessons-so-far"&gt;My lessons so far:&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;Always ask questions: frankly, the only way you can create good content is when you have a solid understanding of the subject matter.&lt;/li&gt;
&lt;li&gt;It's better to over-communicate than under-communicate especially when working in a remotely, this is especially more important if you encounter blockers whilst executing your work.&lt;/li&gt;
&lt;li&gt;Push that code and open PR quickly and then go ahead and ask for a review don't procrastinate this will ensure  fast turnover you get feedback quickly and can work on improvements.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;em&gt;Thank you for reading, watch out for the next update which will be posted soon.&lt;/em&gt;&lt;/p&gt;
</content></entry><entry><title>Add Query Using curl Command and Provide Response Samples</title><link href="http://opensource.creativecommons.org/blog/entries/add-query-using-curl-command-and-provide-response-samples/" rel="alternate"></link><updated>2020-09-25T00:00:00Z</updated><author><name>ariessa</name></author><id>urn:uuid:2841326a-a1a6-3593-8236-10fc75b59aa0</id><content type="html">&lt;p&gt;First of all, I’m very thankful to get selected as a Google Season of Docs participant under Creative Commons. My project name is Improve CC Catalog API Usage Guide. The project aims to revamp the existing CC Catalog API documentation to include more narrative elements and increase user friendliness. As the focal point of this project will potentially be delivered before the end of the GSOD period, this project will also improve the CC Catalog API repo documentation for potential contributors. This project will also produce guidelines for contributing to documentation. For this project, my mentor is Alden Page.&lt;/p&gt;
&lt;h3 id="week-1"&gt;Week 1&lt;/h3&gt;&lt;p&gt;So, the first two weeks of Google Season of Docs have passed. For the first week, I added examples to perform the query using curl command. I hit some problem with a Forbidden error. Turns out my access key got expired. My problem was solved after obtaining a new access key.&lt;/p&gt;
&lt;h3 id="week-2"&gt;Week 2&lt;/h3&gt;&lt;p&gt;For the second week, I started to write response samples. It was tough as I have a hard time understanding &lt;a href="https://github.com/axnsan12/drf-yasg"&gt;drf-yasg&lt;/a&gt;, which is an automatic Swagger generator. It can produce Swagger / OpenAPI 2.0 specifications from a Django Rest Framework API. I tried to find as many examples as I could to increase my understanding. Funny, but it took me awhile to realise that drf-yasg is not made up of random letters. The DRF part stands for Django Rest Framework while YASG stands for Yet Another Swagger Generator.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;That’s all!&lt;/p&gt;
</content></entry><entry><title>The specifics - Revamping CCOS</title><link href="http://opensource.creativecommons.org/blog/entries/the-specifics-revamping-CCOS/" rel="alternate"></link><updated>2020-09-02T00:00:00Z</updated><author><name>dhruvi16</name></author><id>urn:uuid:013d291c-a7ee-3a05-8aef-87f357322155</id><content type="html">&lt;p&gt;In this blog, I will be talking about how I managed to use Vocabulary ( Creative Commons's Design Library ) efficiently in our Open Source website.&lt;/p&gt;
&lt;h3 id="what-is-vocabulary"&gt;What is Vocabulary?&lt;/h3&gt;&lt;p&gt;&lt;a href="https://cc-vocabulary.netlify.app/?path=/story/vocabulary-introduction--page"&gt;Vocabulary&lt;/a&gt; is a cohesive design system to unite the web-facing Creative Commons. In essence Vocabulary is a component library that uses and extends Bulma CSS library. Vocabulary makes it easier to develop Creative Commons apps while ensuring a consistently familiar experience. This project is still under development.&lt;/p&gt;
&lt;h3 id="why-vocabulary"&gt;Why Vocabulary?&lt;/h3&gt;&lt;p&gt;Vocabulary is used to describe the overall visual design of our digital products. At first glance, it appears to be: an amalgamation of component designs with a consistent visual aesthetic and brand, typically accompanied by usage guidelines in the form of online documentation. But there is a lot more to it.
When it comes to a large software community with a huge range of products, certain problems come along. One of those problems is maintaining the level of harmony across all the products of the network. So, there comes a need for a unified visual language that heightens the level of harmony in a digital ecosystem. And in our case, Vocabulary solves this problem.
This design system is well built and helps us bring the following aspects to the table -&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Recognizability&lt;/li&gt;
&lt;li&gt;Consistency&lt;/li&gt;
&lt;li&gt;Authenticity&lt;/li&gt;
&lt;li&gt;Efficiency&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;And many more.&lt;/p&gt;
&lt;h3 id="how-did-i-use-it-examples"&gt;How did I use it? — Examples&lt;/h3&gt;&lt;p&gt;As I stated before, I have added Vocabulary by updating all the Templates in the CCOS &lt;a href="https://www.getlektor.com/"&gt;Lektor&lt;/a&gt; project.&lt;/p&gt;
&lt;p&gt;As far as components are concerned, I just had to paste the code snippets given on the Vocabulary’s website with the requires changes -&lt;/p&gt;
&lt;h4 id="integration-of-breadcrumb"&gt;Integration of Breadcrumb -&lt;/h4&gt;&lt;figure style="text-align: center;"&gt;
    &lt;img src="breadcrumb.png" alt="Breadcrumb"&gt;
    &lt;figcaption&gt;Screenshot — &lt;a href="https://cc-vocabulary.netlify.app/?path=/docs/navigation-breadcrumb--default-story"&gt;Breadcrumb&lt;/a&gt; (Vocabulary)&lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;The code for integration —&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;&amp;lt;!-- Breadcrumb --&amp;gt;
  {% if this._path !=  '/'%}
    &amp;lt;div class="breadcrumb-container"&amp;gt;
      &amp;lt;nav class="container breadcrumb caption bold" aria-label="breadcrumbs"&amp;gt;
        &amp;lt;ul&amp;gt;
          {% set crumbs = [] %}
          {% set current = {'crumb': this} %}
          &amp;lt;!-- Extracting the slugs of URL --&amp;gt;
          {% for i in this._path.split("/") %}
            {% if current.crumb is not none %}
              {% if crumbs.insert(0, current.crumb._slug) %}{% endif %}
              {% if current.update({"crumb": current.crumb.parent}) %}{% endif %}
            {% endif %}
          {% endfor %}
          {% for crumb in crumbs %}
            &amp;lt;!-- Active link --&amp;gt;
            {% if this._slug == crumb %}
              &amp;lt;li class="is-active"&amp;gt;&amp;lt;a aria-current="page displayed"&amp;gt;{{ crumb | title | replace('-', ' ') }}&amp;lt;/a&amp;gt;&amp;lt;/li&amp;gt;
            {% else %}
              &amp;lt;!-- Forming the URL using extracted slugs --&amp;gt;
              {% set i = loop.index %}
              {% set ns = namespace (link = '') %}
              {% for j in range(i) %}
                {% set ns.link = ns.link + crumbs[j] + '/' %}
              {% endfor %}
              &amp;lt;li&amp;gt;&amp;lt;a class="link" href="{{ ns.link|url }}"&amp;gt;
              {% if crumb != '' %}
                {{ crumb | title | replace('-', ' ') }}
              {% else %}
                Home
              {% endif %}
              &amp;lt;/a&amp;gt;&amp;lt;/li&amp;gt;
            {% endif %}
          {% endfor %}
        &amp;lt;/ul&amp;gt;
      &amp;lt;/nav&amp;gt;
    &amp;lt;/div&amp;gt;
  {% endif %}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Other than the components, there are other visual elements like typography, colors, spacing, and others that are extensively used in CCOS.&lt;/p&gt;
&lt;p&gt;This is code for the Hero section of the home page.&lt;/p&gt;
&lt;h5 id="the-block-template"&gt;The block template -&lt;/h5&gt;&lt;pre&gt;&lt;code&gt;&amp;lt;section class="hero"&amp;gt;
  &amp;lt;div class="container"&amp;gt;
    &amp;lt;div class="hero-title column is-12 is-paddingless"&amp;gt;
      &amp;lt;h1&amp;gt;
        {{ this.title }}
      &amp;lt;/h1&amp;gt;
    &amp;lt;/div&amp;gt;
    &amp;lt;div class="columns"&amp;gt;
      &amp;lt;div class="column is-5"&amp;gt;
        &amp;lt;p class="hero-description"&amp;gt;
          {{ this.description }}
        &amp;lt;/p&amp;gt;
        {{ this.links }}
      &amp;lt;/div&amp;gt;
    &amp;lt;/div&amp;gt;
  &amp;lt;/div&amp;gt;
  &amp;lt;div class="level-right hero-image"&amp;gt;
    &amp;lt;img class="image" src="./github.svg" /&amp;gt;
  &amp;lt;/div&amp;gt;
&amp;lt;/section&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;h5 id="the-block-styling"&gt;The block styling -&lt;/h5&gt;&lt;pre&gt;&lt;code&gt;// Hero section - Home page
.hero {
  @extend .margin-top-large;

  .hero-title {
    @extend .padding-horizontal-big;
  }

  .hero-description {
    @extend .body-bigger;
    @extend .padding-top-big;
    @extend .padding-horizontal-big;
  }

  .hero-links {
    @extend .margin-vertical-normal;
    @extend .padding-horizontal-big;

    .button {
      @extend .margin-top-normal;
      text-decoration: none;

      .icon {
        @extend .margin-right-small;
        @extend .padding-vertical-smaller;
      }
    }
  }

  .hero-image {
    @include from($fullhd) {
      margin-top: -20rem;
      .image {
        width: 50%;
      }
    }
    @include until($fullhd) {
      .image {
        width: 100%;
      }
    }
  }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;figure&gt;
    &lt;img src="output.png" alt="Output"&gt;
    &lt;figcaption&gt;Output&lt;/figcaption&gt;
&lt;/figure&gt;&lt;h3 id="improvements-in-the-lektor-project"&gt;Improvements in the Lektor project -&lt;/h3&gt;&lt;p&gt;I tried to write the perfect code that is cleaner and readable. I would try to demonstrate my effort using the home page code where I used &lt;a href="https://www.getlektor.com/docs/models/flow/"&gt;Lektor Flowblocks&lt;/a&gt;. The new homepage design have four sections where each section communicated something and I realized they were all independent and building the whole page through one single template would become a bit messy and hard to handle. So I did some research and found a way where I could build sub-templates and use them all to develop a single page and Lektor’s flowblocks allowed me to do so. Here is one of the flowblock and if you want to check out the whole working you can go to — &lt;a href="https://github.com/creativecommons/creativecommons.github.io-source"&gt;CCOS Repository&lt;/a&gt;.&lt;/p&gt;
&lt;h4 id="recent-blog-post-block"&gt;Recent Blog Post block -&lt;/h4&gt;&lt;h5 id="the-block-template"&gt;The block Template -&lt;/h5&gt;&lt;pre&gt;&lt;code&gt;{% from "macros/author_name.html" import render_author_name %}

&amp;lt;section class="recent-posts"&amp;gt;
  &amp;lt;div class="container"&amp;gt;
    &amp;lt;div class="level"&amp;gt;
      &amp;lt;h2 class="is-paddingless level-left"&amp;gt;
        {{ this.title }}
      &amp;lt;/h2&amp;gt;
      &amp;lt;span class="level-right"&amp;gt;
        &amp;lt;a class="posts-link" href="/blog"&amp;gt;See all posts &amp;lt;i class="icon angle-right"&amp;gt;&amp;lt;/i&amp;gt;&amp;lt;/a&amp;gt;
      &amp;lt;/span&amp;gt;
    &amp;lt;/div&amp;gt;
    &amp;lt;div class="columns"&amp;gt;
      {% for post in site.query('/blog/entries') %}
        {% if loop.index &amp;lt;= 3 %}
          {% set author = post.parent.parent.children.get('authors').children.get(post.author) %}
          &amp;lt;div class="column is-one-third is-paddingless padding-horizontal-big padding-top-bigger"&amp;gt;
            &amp;lt;article class="card entry-post horizontal no-border blog-entry"&amp;gt;
              &amp;lt;header&amp;gt;
                &amp;lt;figure class="image blog-image"&amp;gt;
                {% if author.about %}
                  {% if author.md5_hashed_email %}
                    &amp;lt;img class="profile" src="https://secure.gravatar.com/avatar/{{ author.md5_hashed_email }}?size=200"
                    alt="gravatar" /&amp;gt;
                  {% endif %}
                {% endif %}
                &amp;lt;/figure&amp;gt;
              &amp;lt;/header&amp;gt;
              &amp;lt;div class="blog-content"&amp;gt;
                &amp;lt;h4 class="b-header"&amp;gt;&amp;lt;a class="blog-title" href="{{ post|url }}"&amp;gt;{{ post.title }}&amp;lt;/a&amp;gt;&amp;lt;/h4&amp;gt;
                &amp;lt;span class="blog-author"&amp;gt;by &amp;lt;a class="author-name" href="{{ author|url }}"&amp;gt;{{ render_author_name(author) }}&amp;lt;/a&amp;gt;
                on {{ post.pub_date|dateformat("YYYY-MM-dd") }}&amp;lt;/span&amp;gt;
                &amp;lt;div class="excerpt"&amp;gt;
                  {{ post.body | excerpt | string | striptags() | truncate(100) }}
                &amp;lt;/div&amp;gt;
              &amp;lt;/div&amp;gt;
            &amp;lt;/article&amp;gt;
          &amp;lt;/div&amp;gt;
        {% endif %}
      {% endfor %}
    &amp;lt;/div&amp;gt;
  &amp;lt;/div&amp;gt;
&amp;lt;/section&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;h5 id="the-block-model"&gt;The block Model -&lt;/h5&gt;&lt;pre&gt;&lt;code&gt;[block]
name = Recent Posts

[fields.title]
label = Title
type = string
&lt;/code&gt;&lt;/pre&gt;
&lt;h5 id="the-block-styling"&gt;The block styling -&lt;/h5&gt;&lt;pre&gt;&lt;code&gt;// Recent-posts section - Home page
.recent-posts {
  background-color: rgba(4, 166, 53, 0.1);

  .container {
    @extend .padding-vertical-xl;
    @extend .padding-horizontal-big;

    .columns {
      @extend .padding-top-bigger;
      @extend .padding-bottom-xl;
    }
  }

  .blog-title {
    @extend .has-color-dark-slate-gray;
  }

  .posts-link {
    @extend .has-color-forest-green;
    @extend .body-normal;

    font-weight: bold;
    line-height: 1.5;
    text-decoration: none;

    .icon {
      @extend .has-color-forest-green;
      @extend .padding-left-small;
    }
  }
}
&lt;/code&gt;&lt;/pre&gt;
&lt;figure&gt;
    &lt;img src="output2.png" alt="Output"&gt;
    &lt;figcaption&gt;Output — Recent blog posts.&lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;I would also like to point out the amazing Query functionality provided by Lektor where you can access the child pages of the root. Here I am accessing blog posts from our Blog page and limiting the count of posts to three.&lt;/p&gt;
&lt;h3 id="difference-in-experience"&gt;Difference in Experience&lt;/h3&gt;&lt;p&gt;The level of user experience has been significantly elevated due to the use of Vocabulary. I would like to point one of the major experience change here. The major part of the website is guidelines — we have guidelines for contributing, guidelines for how to join a community, guidelines for how to write a blog and many more. The new website has cleaner and readable guideline with a proper hierarchy and every piece of information is made accessible using secondary navigation.&lt;/p&gt;
&lt;h5 id="below-are-the-images-of-some-guidelines-pages-from-new-website"&gt;Below are the images of some guidelines pages from new website.&lt;/h5&gt;&lt;figure&gt;
    &lt;img width="300" height="300" src="new1.png" alt="Screenshot"&gt;
    &lt;img width="300" height="300" src="new2.png" alt="Screenshot"&gt;
    &lt;img width="300" height="300" src="new3.png" alt="Screenshot"&gt;
    &lt;figcaption&gt;Screenshots from new website&lt;/figcaption&gt;
&lt;/figure&gt;&lt;h5 id="below-are-the-images-of-some-guidelines-pages-from-old-website-you-can-see-the-difference-of-experience-in-both-cases"&gt;Below are the images of some guidelines pages from old website. You can see the difference of experience in both cases.&lt;/h5&gt;&lt;figure&gt;
    &lt;img width="400" src="old1.png" alt="Screenshot"&gt;
    &lt;img width="400" src="old2.png" alt="Screenshot"&gt;
    &lt;figcaption&gt;Screenshots from old website&lt;/figcaption&gt;
&lt;/figure&gt;&lt;h3 id="how-you-can-use-vocabulary-and-also-contribute-to-it"&gt;How you can use Vocabulary and also contribute to it?&lt;/h3&gt;&lt;p&gt;Vocabulary is very easy to use. It is intuitive, consistent and highly reusable. Vocabulary uses Storybook to present each visual element that makes it very convenient for a user to integrate Vocabulary in their project. The code snippets attached with every element can be copied as it is and can be used. The code snippets above indicate how the library can be used and how easily you can achieve desired web pages. For more details, you can visit &lt;a href="https://cc-vocabulary.netlify.app/?path=/docs/vocabulary-usage--page"&gt;usage guidelines&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Vocabulary is still under development, feedback and bug reports are welcome, fixes and patches even more so. Here is the link to &lt;a href="https://cc-vocabulary.netlify.app/?path=/docs/vocabulary-contribution--page"&gt;contribution guidelines&lt;/a&gt;.&lt;/p&gt;
</content></entry><entry><title>Accessibility and Internationalization: WrapUp GSoC 2020</title><link href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-wrapup/" rel="alternate"></link><updated>2020-08-31T00:00:00Z</updated><author><name>AyanChoudhary</name></author><id>urn:uuid:8dd9da39-50da-34b2-91ac-bbe3eee35aee</id><content type="html">&lt;p&gt;These is the final blog of my internship with CC. I am working on improving the accessibility of cc-search and internationalizing it as well.
This blog is the conclusion of my work. These past 10 weeks with CC have taught me a lot and I am really grateful to have got this opportunity.
The experience was just amazing and the poeple are so helpful I really enjoyed working with them and am looking forward to continue working with the CC team.&lt;/p&gt;
&lt;p&gt;You can glance through my work through these blog posts:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-and-internationalization/"&gt;CC Search, Proposal Drafting and Community Bonding&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week1-2/"&gt;CC Search, Setting up vue-i18n and internationalizing homepage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week3-4/"&gt;Internationalization Continued: Handling strings in the store&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week5-6/"&gt;Internationalization continued: Modifying tests&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week7-8/"&gt;CC Search, Initial Accessibility Improvements&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week9-10/"&gt;Accessibility Improvements: Final Changes and Modal Accessilibity&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The progress of the project can be tracked on &lt;a href="https://github.com/creativecommons/cccatalog-frontend"&gt;cc-search&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;CC Search Accessiblity is my GSoC 2020 project under the guidance of &lt;a href="https://creativecommons.org/author/zackcreativecommons-org/"&gt;Zack Krida&lt;/a&gt; and &lt;a href="https://opensource.creativecommons.org/blog/authors/akmadian/"&gt;Ari Madian&lt;/a&gt;, who is the primary mentor for this project, &lt;a href="https://creativecommons.org/author/annacreativecommons-org/"&gt;Anna Tumadóttir&lt;/a&gt; for helping all along and engineering director &lt;a href="https://creativecommons.org/author/kriticreativecommons-org/"&gt;Kriti
Godey&lt;/a&gt;, have been very supportive.&lt;/p&gt;
</content></entry><entry><title>Linked Commons: GSoC'20 Wrap Up</title><link href="http://opensource.creativecommons.org/blog/entries/linked-commons-gsoc-wrap-up/" rel="alternate"></link><updated>2020-08-28T00:00:00Z</updated><author><name>subhamX</name></author><id>urn:uuid:511904ae-a464-335a-bcdf-76f80c5309d1</id><content type="html">&lt;p&gt;Time flies faster when you are having fun! I didn't believe it back then. But now I do after experiencing it. It couldn't have been more accurate that here I am writing my concluding blog of the &lt;strong&gt;GSoC 2020: The Linked Commons series&lt;/strong&gt; when I just started enjoying things.&lt;/p&gt;
&lt;p&gt;In this post, I will give a brief overview of the linked commons and my GSoC contributions. It was an exciting journey, and I loved working on this project.&lt;/p&gt;
&lt;p&gt;Before I begin any further just for ritual, let me share a one liner on &lt;strong&gt;what The Linked Commons is&lt;/strong&gt;, although I highly recommend reading the other posts in this series, who knows you might join our team.😉&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;The CC catalog data visualization or linked commons is a web application which finds and explores relationships between Creative Commons licensed content on the web.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;My primary contributions to the project during the GSoC timeline were threefold.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Firstly&lt;/strong&gt;, revamp the design and migrate the project to react.js for a fast and scalable rendering performance.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Secondly&lt;/strong&gt;, add graph filtering methods and scale the data to enable users to visualize massive data more efficiently.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;At last&lt;/strong&gt;, make the developer onboarding easy by dockerizing the project and bring more application portability.&lt;/p&gt;
&lt;h2 id="gsoc-work-product"&gt;GSoC Work Product&lt;/h2&gt;&lt;p&gt;The live version of the linked commons can be found &lt;a href="http://dataviz.creativecommons.engineering/"&gt;here&lt;/a&gt;. You can interact with it and &lt;strong&gt;"explore the creative commons in graphs"&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;If you wish to access the raw or filtered data, then here is a brief documentation of our new API.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nt"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/api/graph-data&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="nt"&gt;Method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;GET&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="nt"&gt;Description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Returns a randomized graph having around 500 nodes and links.&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nt"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/api/graph-data/?name={node_name}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="nt"&gt;Method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;GET&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="nt"&gt;Description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Returns the filtered graph with a set of nodes which are either immediate neighbours to {node_name} in the original graph or the transpose graph.&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&amp;nbsp;&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nt"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/api/suggestions/?q={query}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="nt"&gt;Method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;GET&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;
&lt;span class="nt"&gt;Description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Returns a set of nodes which contains the {query} pattern in their nodeid&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;h3 id="demo"&gt;Demo&lt;/h3&gt;&lt;div style="text-align: center; width: 90%; margin-left: 5%;"&gt;
    &lt;figure&gt;
        &lt;img src="graph-filtering.gif" alt="demo" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;Linked Commons: Filtering the Graph 🔥&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;h3 id="my-code-contributions"&gt;My Code Contributions&lt;/h3&gt;&lt;p&gt;&lt;strong&gt;Repository:&lt;/strong&gt; &lt;a href="https://github.com/creativecommons/cccatalog-dataviz/"&gt;https://github.com/creativecommons/cccatalog-dataviz/&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Commits:&lt;/strong&gt; &lt;a href="https://github.com/creativecommons/cccatalog-dataviz/commits/master"&gt;https://github.com/creativecommons/cccatalog-dataviz/commits/master&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Contributors:&lt;/strong&gt; &lt;a href="https://github.com/creativecommons/cccatalog-dataviz/graphs/contributors"&gt;https://github.com/creativecommons/cccatalog-dataviz/graphs/contributors&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href="https://github.com/creativecommons/cccatalog-dataviz/pull/28"&gt;&lt;strong&gt;Migrate frontend to React #28&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Migrated the frontend to a web application using React.js for smooth rendering performance.&lt;/li&gt;
&lt;li&gt;Add client-side graph filtering method to enable users to interact with the loaded graph.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;a href="https://github.com/creativecommons/cccatalog-dataviz/pull/29"&gt;&lt;strong&gt;Add server-side filtering #29&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;We realized that the client-side graph filtering method is not very scalable. This PR adds the basic structure for the backend server and adds server-side graph filtering logic.&lt;/li&gt;
&lt;li&gt;Added a parser to convert the input JSON file from &lt;code&gt;{nodes:[], links:[]}&lt;/code&gt; schema to the distance list format. &lt;/li&gt;
&lt;/ul&gt;
&lt;div style="text-align: center; width: 90%; margin-left: 5%;"&gt;
    &lt;figure&gt;
        &lt;img src="api-call.png" alt="API call" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;API call to filter graph data&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;p&gt;&lt;a href="https://github.com/creativecommons/cccatalog-dataviz/pull/33"&gt;&lt;strong&gt;Design upgrade #33&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;It revamped the design of the frontend.&lt;/li&gt;
&lt;li&gt;Added both primary light theme and secondary dark theme&lt;/li&gt;
&lt;/ul&gt;
&lt;div style="text-align: center; font-style: normal; width: 80%; margin-left: 10%;"&gt;
    &lt;figure&gt;
        &lt;img src="design-dark.png" alt="Dark Theme" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;&lt;em&gt;Linked Commons: Dark Theme&lt;/em&gt;&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;p&gt;&lt;a href="https://github.com/creativecommons/cccatalog-dataviz/pull/35"&gt;&lt;strong&gt;Add node suggestions feature #35&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Added query autocomplete feature, to enable users to explore all the nodes in the database.&lt;/li&gt;
&lt;li&gt;This functionality aims to minimize the number of misspelt filtering tries from the client. &lt;/li&gt;
&lt;li&gt;Refer to &lt;a href="/blog/entries/linked-commons-autocomplete-feature/"&gt;this blog&lt;/a&gt; for the motivation and detailed report on why we added autocomplete aka node suggestions feature.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;a href="https://github.com/creativecommons/cccatalog-dataviz/pull/38"&gt;&lt;strong&gt;Fix filtering module #38&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Optimizes the build-dB-script to run efficiently on the larger and newer dataset of the cc-catalog.&lt;/li&gt;
&lt;li&gt;Added the basic form of the randomized graph filtering method.&lt;/li&gt;
&lt;li&gt;Refer to &lt;a href="/blog/entries/linked-commons-data-update/"&gt;this blog&lt;/a&gt; for a piece of detailed information on data update.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;a href="https://github.com/creativecommons/cccatalog-dataviz/pull/39"&gt;&lt;strong&gt;Database upgrade and core enhancements #39&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;It upgrades the primary database from shelve to MongoDB for higher performance.&lt;/li&gt;
&lt;li&gt;Dockerizes the frontend and backend for both dev and prod environments for higher application portability.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;a href="https://github.com/creativecommons/cccatalog-dataviz/pull/40"&gt;&lt;strong&gt;Frontend enhancements #40&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Fixes common UI bugs, updates the frontend design and enhances the mobile and smaller devices experience with the linked commons&lt;/li&gt;
&lt;li&gt;Modularizes and updates the code documentation&lt;/li&gt;
&lt;/ul&gt;
&lt;div style="text-align: center; width: 80%; margin-left: 10%;"&gt;
    &lt;figure&gt;
        &lt;img src="design-light.png" alt="Theme Light" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;Linked Commons: Light Theme&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;div style="text-align: center; width: 90%; margin-left: 5%;"&gt;
    &lt;figure&gt;
        &lt;img src="lighthouse-audit.png" alt="Lighthouse Audit" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;Lighthouse Stats of the latest version of the Linked Commons&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;h3 id="whats-next"&gt;What’s Next?&lt;/h3&gt;&lt;p&gt;Throughout this internship period, we, the Linked Commons team, aimed to make this version the best among all. But there is still scope for improvement.&lt;/p&gt;
&lt;p&gt;Just to give you some insights; currently, the complete graph contains 235k nodes and 4.14million links. During the preprocessing, we dropped a lot of the nodes. Additionally, we removed more than 3 million nodes which didn't have cc_licenses information. So, in general, the current version shows only those nodes which are soundly linked with other domains and their licenses information is available. To give a complete picture of the massive "cc-catalog", the linked commons need to "gird up his loins".&lt;/p&gt;
&lt;p&gt;After seeing the tremendous potential it has, I will undoubtedly continue working on it and help the linked commons in this quest. ⚡&lt;/p&gt;
&lt;h3 id="ending-note"&gt;Ending Note&lt;/h3&gt;&lt;p&gt;In the end, I would like to thank my mentors Maria and Brent, for their unconditional guidance throughout this internship period. The insights I got from them will truly help me in the days to come.&lt;/p&gt;
&lt;p&gt;Special thanks to Francisco, Anna and Kriti for the awesome brainstorming ideas in the UX meet which helped us build an increment superior version of the Linked Commons.&lt;/p&gt;
&lt;p&gt;It is not the end, rather a new beginning. Cheers! 🚀🚀🚀&lt;/p&gt;
</content></entry><entry><title>CC Search Extension: Wrapping up GSoC 2020</title><link href="http://opensource.creativecommons.org/blog/entries/cc-search-extension-wrapping-up-gsoc-2020/" rel="alternate"></link><updated>2020-08-27T00:00:00Z</updated><author><name>makkoncept</name></author><id>urn:uuid:a836222c-bec0-3f28-a8cd-9699e8ba778f</id><content type="html">&lt;p&gt;In this post, I'll give an overview of the improvements and features that were added to the CC Search browser extension. I am delighted to state that the goals that were set for Google Summer of Code 2020 have been successfully completed.&lt;/p&gt;
&lt;h2 id="widen-the-integration-with-cc-catalog-api"&gt;Widen the integration with CC Catalog API&lt;/h2&gt;&lt;p&gt;Both &lt;a href="/blog/entries/cc-search-extension-wrapping-up-gsoc-2020/search.creativecommons.org"&gt;CC Search&lt;/a&gt; and CC Search Extension are powered by &lt;a href="https://api.creativecommons.engineering/v1/"&gt;CC Catalog REST API&lt;/a&gt;. The API allows programmatic access to search for CC-licensed and public domain digital media. Better integration with the API was one of the major targets during this internship because it significantly improves and adds new searching workflows to the extension.&lt;/p&gt;
&lt;p&gt;This can be sub-divided into &lt;em&gt;New Filters&lt;/em&gt;, &lt;em&gt;Browse By Sources&lt;/em&gt;, &lt;em&gt;search by tags&lt;/em&gt;, and &lt;em&gt;related images&lt;/em&gt;.&lt;/p&gt;
&lt;h3 id="new-filters"&gt;New Filters&lt;/h3&gt;&lt;p&gt;The &lt;code&gt;/image&lt;/code&gt; endpoint of the API is used for searching. We can also provide several query parameters that can filter the result. Previously, the extension only supported filtering the content using &lt;code&gt;license&lt;/code&gt;, &lt;code&gt;sources&lt;/code&gt;, and &lt;code&gt;use case&lt;/code&gt;. Now, besides these filters, the extension also supports filtering by &lt;code&gt;image type&lt;/code&gt;, &lt;code&gt;file type&lt;/code&gt;, &lt;code&gt;aspect ratio&lt;/code&gt;, and &lt;code&gt;image size&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="old-extension-filters.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Filters in the old version&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;figure&gt;
    &lt;img src="new-extension-filters.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Filters in the new version&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;&lt;em&gt;Rationale&lt;/em&gt;: This will allow users to be more precise in their queries when searching.&lt;/p&gt;
&lt;h3 id="browsing-by-source"&gt;Browsing by source&lt;/h3&gt;&lt;p&gt;The extension now has a dynamically updated "sources" section. Clicking a source link triggers a request to the &lt;code&gt;/image&lt;/code&gt; endpoint to get the images associated with it.&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="source-section-light.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Source section&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;figure&gt;
    &lt;img src="source-section-dark.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Source section in dark mode&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;&lt;em&gt;Rationale&lt;/em&gt;: This opens an avenue for exploration of all the different sources which are available in the catalog. This is advantageous for the users who are not familiar with the type of content a particular source provides. They might run into a source that has a huge catalog of high-quality images that they are looking for.&lt;/p&gt;
&lt;h3 id="search-by-tags"&gt;Search by tags&lt;/h3&gt;&lt;p&gt;Most of the images have some tags associated with them, which are also sent along with the image data by the API. This, and the flexibility of the &lt;code&gt;/image&lt;/code&gt; endpoint, paved the way for the addition of searching for images using image-tags.&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="search-by-image-tag.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Search by image tag&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Rational&lt;/em&gt; - Image tags will allow users to incrementally make their queries better and more specific.&lt;/p&gt;
&lt;h3 id="related-images"&gt;Related images&lt;/h3&gt;&lt;p&gt;In the image detail section of any particular image, you can now see several recommendations. This has been made possible by adding support for the &lt;a href="https://api.creativecommons.engineering/v1/#tag/recommendations"&gt;&lt;code&gt;/recommendations/images/{identifier}&lt;/code&gt;&lt;/a&gt; endpoint of the API.&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="related-images.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Image recommendations&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Rationale&lt;/em&gt; -  This will help users find a variety of images that fit their requirements and also explore the images that would not usually show up on the initial pages of the search result.&lt;/p&gt;
&lt;h2 id="improvements-to-bookmarks-section"&gt;Improvements to bookmarks section&lt;/h2&gt;&lt;p&gt;The bookmark section has great prominence in CC Search Extension because the export/import workflow is tied to it and unlike the search result data, the bookmarks data is preserved across user sessions (closing the extension does not wipe out the bookmarks). It has undergone some crucial improvements like caching, voluntary loading and increase in the number bookmarks that it can hold (the limit now is 300 which earlier was ~50).&lt;/p&gt;
&lt;p&gt;The bookmarks section is significantly faster now as caching has eliminated the need to make many simultaneous network requests to the API when bookmarks are loaded. Voluntary loading also helps reduce perceived lag by reducing the number of bookmarks that load at once.&lt;/p&gt;
&lt;p&gt;Though the improvement in performance is better recognized when you are using the extension, I tried to demonstrate that by comparing the rendering of the bookmarked images.&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="bookmarks-in-old-version.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Bookmark section in the old version&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="bookmarks-in-new-version.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Bookmarks section in the new version&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;h2 id="a-better-use-of-sync-storage"&gt;A better use of sync storage&lt;/h2&gt;&lt;p&gt;The bookmarks and the user settings are synced between user systems. There are very tight write limits and bytes quotas associated with this storage(&lt;a href="https://developer.chrome.com/apps/storage#properties"&gt;documentation link&lt;/a&gt;). Due to this, the way the extension used this storage, and the assumptions it made about its schema was improved multiple times. Since the extension was already in production, and had around 5,000 weekly users, the code for migrating the user's sync storage was pushed along with these updates. Also, the support was added for legacy bookmark files that some users might still be using.&lt;/p&gt;
&lt;h2 id="integration-with-vocabulary"&gt;Integration with Vocabulary&lt;/h2&gt;&lt;p&gt;The extension now supports the latest version of &lt;a href="https://github.com/creativecommons/vocabulary"&gt;CC vocabulary&lt;/a&gt;. The challenging part of this was to rethink, mold, and update each and every workflow of the extension according to the new design.&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="image-detail-old-version.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Old version — Image detail&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="image-detail-new-version.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;New version — Image detail&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="deletion-in-old-version.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Old version — Deleting bookmarks&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="deletion-in-new-version.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;New version — Deleting bookmarks&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="dark-mode-old-version.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;Old version — Dark mode&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;p&gt;&lt;figure &gt;
    &lt;img src="dark-mode-new-version.gif" style="width: 70%"&gt;
    &lt;figcaption&gt;
        &lt;em&gt;New version — Dark mode&lt;/em&gt;
    &lt;/figcaption&gt;
&lt;/figure&gt;&lt;/p&gt;
&lt;h2 id="release-on-microsoft-edge"&gt;Release on Microsoft Edge&lt;/h2&gt;&lt;p&gt;I am also testing the extension on Microsoft Edge. We also have it &lt;a href="https://microsoftedge.microsoft.com/addons/detail/cc-search/djolilnbndifmlfmcdnifdfjfbglipgc"&gt;listed&lt;/a&gt; on the Edge store. You can soon expect the latest version of CC Search Extension available for install there.&lt;/p&gt;
&lt;h2 id="code"&gt;Code&lt;/h2&gt;&lt;p&gt;The project repository is hosted on &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension"&gt;Github&lt;/a&gt;. During this period, I have made &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/compare/v1.3.0...master"&gt;more than 320&lt;/a&gt; commits.&lt;/p&gt;
&lt;p&gt;The Major pull requests: &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/pull/249"&gt;#249&lt;/a&gt;, &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/pull/255"&gt;#255&lt;/a&gt;, &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/pull/268"&gt;#268&lt;/a&gt;, &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/pull/270"&gt;#270&lt;/a&gt;, &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/pull/271"&gt;#271&lt;/a&gt;, &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/pull/272"&gt;#272&lt;/a&gt;, &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/pull/275"&gt;#275&lt;/a&gt;, &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/pull/276"&gt;#276&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;Also, during this period, 5 updates of the extension were pushed to the extension stores. You can check out the &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension/releases"&gt;releases page&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id="acknowledgements"&gt;Acknowledgements&lt;/h2&gt;&lt;p&gt;I would like to thank &lt;a href="https://creativecommons.org/author/aldencreativecommons-org/"&gt;Alden&lt;/a&gt; and &lt;a href="https://creativecommons.org/author/kriticreativecommons-org/"&gt;Kriti&lt;/a&gt; for their valuable guidance during this journey. Special thanks to &lt;a href="https://github.com/panchovm"&gt;Fransisco&lt;/a&gt;, for designing the mockups of the extension, and to the wonderful contributors of CC Vocabulary.&lt;/p&gt;
</content></entry><entry><title>Automate GitHub for more than CI/CD</title><link href="http://opensource.creativecommons.org/blog/entries/automate-github-for-more-than-CI%20CD/" rel="alternate"></link><updated>2020-08-26T00:00:00Z</updated><author><name>zackkrida</name></author><id>urn:uuid:1f2b4dad-de07-33ab-b1c7-394778548e55</id><content type="html">&lt;blockquote&gt;&lt;p&gt;&lt;em&gt;Get started using GitHub bots and actions for community management and repository health.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;In late 2018, in the midst of being acquired by Microsoft, GitHub &lt;a href="https://github.blog/2018-10-16-future-of-software/"&gt;launched Github Actions&lt;/a&gt; into public beta, allowing users to run code on the popular development platform for the first time. With a straightforward &lt;code&gt;YAML&lt;/code&gt; configuration syntax and the power of Microsoft's Azure cloud, GitHub Actions quickly rose to compete with existing Continuous Integration (CI) and Continuous Deployment (CD) platforms like &lt;strong&gt;Circle CI&lt;/strong&gt; and &lt;strong&gt;Travis CI&lt;/strong&gt;. GitHub Actions made it easier than ever for developers to test and deploy software in the cloud, but from the beginning GitHub had bigger plans for the service.&lt;/p&gt;
&lt;p&gt;In a &lt;a href="https://techcrunch.com/2018/10/16/github-launches-actions-its-workflow-automation-tool/"&gt;2018 TechCrunch interview&lt;/a&gt;, GitHub's then head of platform acknowledged the usefulness of actions for more than CI/CD. "I see CI/CD as one narrow use case of actions. It’s so, so much more,” Lambert stressed. “And I think it’s going to revolutionize DevOps because people are now going to build best in breed deployment workflows for specific applications and frameworks, and those become the de facto standard shared on GitHub. […] It’s going to do everything we did for open source again for the DevOps space and for all those different parts of that workflow ecosystem."&lt;/p&gt;
&lt;p&gt;At Creative Commons, we use Github Actions and Bots on many of &lt;a href="https://github.com/creativecommons?type=source"&gt;our open-source projects&lt;/a&gt; for more than CI/CD—to manage our &lt;a href="/community/community-team/"&gt;community team&lt;/a&gt;; to automate repository health; and to automate tedious but frequent tasks. The following examples are just a small snapshot of our existing and in-progress automations.&lt;/p&gt;
&lt;h2 id="example-automations"&gt;Example automations&lt;/h2&gt;&lt;p&gt;&lt;!-- no toc --&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="/blog/entries/automate-github-for-more-than-CI CD/#automatic-release-note-generation"&gt;Release note generation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="/blog/entries/automate-github-for-more-than-CI CD/#repository-normalization"&gt;Repository normalization&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="/blog/entries/automate-github-for-more-than-CI CD/#automatic-dependency-updates"&gt;Dependency updates&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="release-note-generation"&gt;Release note generation&lt;/h3&gt;&lt;p&gt;Our frontend Vue.js application for CC Search gets released weekly, and is subject to constant pull requests from myself, one-time volunteers making their first open source contribution, and long-term, dedicated community members who frequently contribute. It's important for us to highlight &lt;em&gt;all&lt;/em&gt; of these contributions in our release notes, regardless of size or scope. Additionally, we find it useful to group changes into categories, so our users have a clear sense of what kinds of updates we've made.&lt;/p&gt;
&lt;div style="text-align: center;"&gt;
  &lt;figure class="margin-bottom-large"&gt;
    &lt;img src="release-notes-screenshot.png" alt="GitHub screenshot of release notes for CC Search" /&gt;
    &lt;figcaption&gt;
      &lt;em&gt;
        An example of CC Search release notes generated by the &lt;a href="https://github.com/marketplace/actions/release-drafter"&gt;Release Drafter&lt;/a&gt; GitHub Action.
      &lt;/em&gt;
    &lt;/figcaption&gt;
  &lt;/figure&gt;
&lt;/div&gt;&lt;p&gt;The quality of these release notes made them quite tedious to generate manually. With the &lt;a href="https://github.com/marketplace/actions/release-drafter"&gt;release drafter action&lt;/a&gt;, we're able to automatically update a draft release note on every pull request to CC Search. The action lets us configure the line added for each pull request with some basic templating which includes variables for the pr number, title, and author (among others):&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nt"&gt;change-template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;-&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;$TITLE:&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;#$NUMBER&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;by&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;@$AUTHOR&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;br /&gt;This means each pull request gets a line like this in our release notes:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;Enable web monetization on single result pages: &lt;strong&gt;#1191&lt;/strong&gt; by &lt;strong&gt;@zackkrida&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Perfect! We can also map GitHub labels on our pull requests to the sections of our generated release notes, like so:&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nt"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;New&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Features&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;label&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;feature&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;Bug&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Fixes&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;label&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;bug&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;critical&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The resulting release notes require no manual editing at release time, and has saved us hours over time and allows our developers to focus on DevOps work instead of copywriting on release days. We also never miss a contribution or expression of gratitude to one of our contributors. You can read the &lt;a href="https://github.com/creativecommons/cccatalog-frontend/releases/latest"&gt;latest CC Search release notes&lt;/a&gt; or &lt;a href="https://github.com/creativecommons/cccatalog-frontend/blob/develop/.github/release-drafter.yml"&gt;see our full release-drafter.yml file here&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id="repository-normalization"&gt;Repository Normalization&lt;/h3&gt;&lt;p&gt;Within a private repository of internal helper scripts, the CC technical team has a number of Github Actions which trigger Python scripts to keep configuration standardized across our repositories. We casually call this process "repository normalization". One such script ensures that we use a standard set of GitHub labels across all of our projects. This consistency helps us do things like direct users to &lt;a href="https://github.com/search?q=org%3Acreativecommons+label%3A%22help+wanted%22+state%3Aopen&amp;amp;type=Issues"&gt;open issues in need of assistance&lt;/a&gt; across the organization, or issues &lt;a href="https://github.com/search?q=org%3Acreativecommons+label%3A%22good+first+issue%22+state%3Aopen&amp;amp;type=Issues"&gt;good for first-time open source contributors&lt;/a&gt;. With GitHub Actions, its easy to set up scheduled tasks with only a few lines of human-readable configuration. Here's the gist of running a Python script daily, for example:&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Example scheduled python action&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="nt"&gt;on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;cron&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;0&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;push&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;branches&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;master&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="nt"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;build&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ubuntu-latest&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;actions/checkout@v2&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Set up Python 3.7&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;actions/setup-python@v1&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;python-version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;3.7&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Install dependencies&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;python -m pip install --upgrade pip&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;python -m pip install pipenv&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;pipenv install&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Export token to env and run our script&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;pipenv run python our-script.py&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;ADMIN_GITHUB_TOKEN&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;${{ secrets.ADMIN_GITHUB_TOKEN }}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Internally and publicly, we use &lt;a href="https://github.com/orgs/creativecommons/projects"&gt;GitHub Projects&lt;/a&gt; to manage our bi-weekly sprints and backlogs. The &lt;a href="https://github.com/subhamX/github-project-bot"&gt;GitHub Project Bot&lt;/a&gt; action was built by &lt;a href="https://github.com/subhamX"&gt;one of our community contributors&lt;/a&gt; and allows us to add pull requests to our project columns. Here's an example step in such a job:&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Handle cccatalog-frontend Repo&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;subhamX/github-project-bot@v1.0.0&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;ACCESS_TOKEN&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;${{ secrets.ADMIN_GITHUB_TOKEN }}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;COLUMN_NAME&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;In&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Progress&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;(Community)&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;PROJECT_URL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;https://github.com/orgs/creativecommons/projects/7&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;REPO_URL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;https://github.com/creativecommons/cccatalog-frontend&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;We have additional scripts that sync our community team members across our open source website and GitHub, and several others that do even more of this cross-platform synchronization work. All of these scripts relive significant burden off of our engineering manager and open source community coordinator.&lt;/p&gt;
&lt;h3 id="dependency-updates"&gt;Dependency Updates&lt;/h3&gt;&lt;p&gt;Modern JavaScript projects are built atop piles of 3rd party dependencies. This frees developers to focus on product code instead of writing the same utility code over and over again, but exposes projects to issues of security and dependency management. To help alleviate these issues, GitHub &lt;a href="https://github.blog/2019-05-23-introducing-new-ways-to-keep-your-code-secure/#automated-security-fixes-with-dependabot"&gt;acquired a startup called Dependabot&lt;/a&gt; which initially focused on automatic security updates for repositories. Dependabot creates pull requests that update third-party code  with known security vulnerabilities to the latest safe and stable versions.&lt;/p&gt;
&lt;p&gt;This summer (June 2020), GitHub &lt;a href="https://github.blog/2020-06-01-keep-all-your-packages-up-to-date-with-dependabot/"&gt;expanded dependabot's scope&lt;/a&gt; to keep &lt;em&gt;all&lt;/em&gt; third-party code up to date, regardless of security. By adding a &lt;code&gt;dependabot-config.yml&lt;/code&gt; file to any repo, developers no longer need to keep track of dependency updates on their own.&lt;/p&gt;
&lt;div style="text-align: center;"&gt;
  &lt;figure class="margin-bottom-large"&gt;
    &lt;img src="dependabot-example.png" alt="GitHub screenshot of a Dependabot PR message" /&gt;
    &lt;figcaption&gt;
      &lt;em&gt;
        Dependabot writes pull requests to bump JavaScript dependencies and will automatically resolve merge conflicts and keep the PR up to date.
      &lt;/em&gt;
    &lt;/figcaption&gt;
  &lt;/figure&gt;
&lt;/div&gt;&lt;p&gt;If your project has strong test coverage and a solid quality control process for release management, Dependabot pull requests can be made even more powerful with the &lt;a href="https://github.com/ridedott/merge-me-action"&gt;Merge Me Action.&lt;/a&gt; Merge Me can be added to the end of any series of Github Actions to automatically merge pull requests that pass all CI tests which were authored by a particular user (the action assumes &lt;code&gt;dependabot&lt;/code&gt; by default). This means your repository can have highly-configurable, fully-automated dependency updates in just a few lines of &lt;code&gt;YAML&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="here-s-a-few-more"&gt;Here's a few more&lt;/h2&gt;&lt;p&gt;Here's some smaller and simpler automations that can make a huge difference in your workflows.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/probot/stale"&gt;Automatically close old PRs after a period of inactivity&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.blog/2020-08-24-automate-releases-and-more-with-the-new-sentry-release-github-action/"&gt;Automate security releases on Sentry&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/probot/reminders"&gt;Add reminders to issues and pull requests&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These examples are a small sample of the non-CI/CD capabilities of GitHub Actions. You can peek in the &lt;code&gt;.github/&lt;/code&gt; directory of any of our open source repositories to see the actions we're using, and feel free to make an issue on any project if you have an idea for an automation of your own. As we increase the number and quality of integrations in our open source repositories, we may update this article or create follow-up posts with more examples.&lt;/p&gt;
&lt;p&gt;If you're interested in learning more about GitHub Actions, GitHub has a wonderful &lt;a href="https://github.com/marketplace?type=actions"&gt;marketplace&lt;/a&gt; of avaliable actions you can explore, and the &lt;a href="https://docs.github.com/actions"&gt;documentation for actions&lt;/a&gt; is avaliable in several languages.&lt;/p&gt;
</content></entry><entry><title>Overview of the GSoC 2020 Project</title><link href="http://opensource.creativecommons.org/blog/entries/overview-of-the-gsoc-2020-project/" rel="alternate"></link><updated>2020-08-26T00:00:00Z</updated><author><name>charini</name></author><id>urn:uuid:4cdfc111-c714-33e1-b521-390d487d46da</id><content type="html">&lt;p&gt;This is my final blog post under the &lt;a href="/blog/entries/overview-of-the-gsoc-2020-project/#series"&gt;GSoC 2020: CC catalog&lt;/a&gt; series, where I will highlight and
summarize my contributions to Creative Commons (CC) as part of my GSoC project. The CC Catalog project collects and
stores CC licensed images scattered across the internet, such that they can be made accessible to the general public via
the &lt;a href="https://ccsearch.creativecommons.org/"&gt;CC Search&lt;/a&gt; and &lt;a href="https://api.creativecommons.engineering/v1/"&gt;CC Catalog API&lt;/a&gt; tools. I got the opportunity to work on different aspects of the
CC Catalog repository which ultimately enhances the user experience of the CC Search and CC Catalog API tools. My
primary contributions in the duration of GSoC, and the related pull requests (PR) are as follows.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Sub-provider retrieval&lt;/strong&gt;: The first task I completed as part of my GSoC project was the retrieval of sub-providers
(also known as &lt;em&gt;source&lt;/em&gt;) such that images could be categorised under these sources, ensuring an enhanced search
experience for the users. I completed the implementation of sub-provider retrieval for three providers; Flickr,
Europeana, and Smithsonian. If you are interested in learning how the retrieval logic works, please check my
&lt;a href="/blog/entries/flickr-sub-provider-retrieval/"&gt;initial blog post&lt;/a&gt; of this series. The PRs related to this task are as follows.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;PR #&lt;a href="https://github.com/creativecommons/cccatalog/pull/420"&gt;420&lt;/a&gt;: Retrieve sub-providers within Flickr&lt;/li&gt;
&lt;li&gt;PR #&lt;a href="https://github.com/creativecommons/cccatalog/pull/442"&gt;442&lt;/a&gt;: Retrieve sub-providers within Europeana&lt;/li&gt;
&lt;li&gt;PR #&lt;a href="https://github.com/creativecommons/cccatalog/pull/455"&gt;455&lt;/a&gt;: Retrieve sub-providers within Smithsonian&lt;/li&gt;
&lt;li&gt;PR #&lt;a href="https://github.com/creativecommons/cccatalog/pull/461"&gt;461&lt;/a&gt;: Add new source as a sub-provider of Flickr&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Alert updates to Smithsonian unit codes&lt;/strong&gt;: For the Smithsonian provider, we rely on the field known as &lt;em&gt;unit code&lt;/em&gt;
to determine the sub-provider (for Smithsonian it is often a museum) each image belongs to. However, it is possible for
the &lt;em&gt;unit code&lt;/em&gt; values to change over time at the upstream, and if CC is unaware of these changes, it could hinder the
successful categorisation of Smithsonian images under unique sub-provider values. I have therefore introduced a
mechanism of alerting the CC code maintainers of potential changes to &lt;em&gt;unit code&lt;/em&gt; values at the upstream. More
information is provided in my &lt;a href="/blog/entries/smithsonian-unit-code-update/"&gt;second blog post&lt;/a&gt; of this series. The PR related to this task
is #&lt;a href="https://github.com/creativecommons/cccatalog/pull/465"&gt;465&lt;/a&gt;.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Improvements to the Smithsonian provider API script&lt;/strong&gt;: Smithsonian is an important provider which aggregates images
from 19 museums. However, due to the fact that the different museums have different data models and the resultant
incompatibility of the JSON responses returned from requests to the Smithsonian API, it is difficult to know which
fields to rely on to obtain the information necessary for CC. This results in CC missing out on certain important
information. As part of my GSoC project, I improved the completeness of &lt;em&gt;creator&lt;/em&gt; and &lt;em&gt;description&lt;/em&gt; information, by
identifying previously unknown fields from which these details could be retrieved. Even though my improvements did not
result in the identification of a comprehensive list of fields, the completeness of data was considerably improved for
some Smithsonian museums compared to how it was before. For more context about this issue please refer to the ticket
#&lt;a href="https://github.com/creativecommons/cccatalog/issues/397"&gt;397&lt;/a&gt;. Apart from improving information of Smithsonian data, I was also able to identify issues with certain
Smithsonian API responses which did not contain mandatory information for some of the museums. We have informed the
Smithsonian technical team of these issues and they are highlighted in ticket #&lt;a href="https://github.com/creativecommons/cccatalog/issues/397"&gt;397&lt;/a&gt; as well. The PRs related
to this task are as follows.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;PR #&lt;a href="https://github.com/creativecommons/cccatalog/pull/474"&gt;474&lt;/a&gt;: Improve the creator and description information of the Smithsonian source &lt;em&gt;National Museum of
Natural History&lt;/em&gt; (NMNH). This is the largest museum (source) under the Smithsonian provider.&lt;/li&gt;
&lt;li&gt;PR #&lt;a href="https://github.com/creativecommons/cccatalog/pull/476"&gt;476&lt;/a&gt;: Improve the &lt;em&gt;creator&lt;/em&gt; and &lt;em&gt;description&lt;/em&gt; information of other sources coming under the Smithsonian
provider.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Expiration of outdated images&lt;/strong&gt;: The final task I completed as part of my GSoC project was implementing a strategy
for expiring outdated images in the CC database. CC has a mechanism for keeping the images they have retrieved from
providers up-to-date, based on how old an image is. This is called the &lt;a href="/blog/entries/date-partitioned-data-reingestion/"&gt;re-ingestion strategy&lt;/a&gt;,
where newer images are updated more frequently compared to older images. However, this re-ingestion strategy does not
detect images which have been deleted at the upstream. Thus, it is possible that some of the images stored in the CC
database are obsolete, which could result in broken links being presented via the &lt;a href="https://ccsearch.creativecommons.org/"&gt;CC Search&lt;/a&gt; tool. As a
solution, I have implemented a mechanism of identifying whether images in the CC database are obsolete by looking at the
&lt;em&gt;updated_on&lt;/em&gt; column value of the CC image table. Depending on the re-ingestion strategy per provider, we can know what
the oldest &lt;em&gt;updated_on&lt;/em&gt; value, an image can assume. If the &lt;em&gt;updated_on&lt;/em&gt; value is older than the oldest valid value, we
flag the corresponding image record  as obsolete.  The PR related to this task is #&lt;a href="https://github.com/creativecommons/cccatalog/pull/483"&gt;483&lt;/a&gt;.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;I will continue to take the responsibility for maintaining my code in the CC Catalog repository, and I hope to continue
contributing to the CC codebase. It has been a wonderful GSoC journey for me and special thanks goes to my supervisor
Brent for his guidance.&lt;/p&gt;
</content></entry><entry><title>Linked Commons: Data Update</title><link href="http://opensource.creativecommons.org/blog/entries/linked-commons-data-update/" rel="alternate"></link><updated>2020-08-25T00:00:00Z</updated><author><name>subhamX</name></author><id>urn:uuid:bf2032b7-d1c2-320b-9cf1-92ad64320a02</id><content type="html">&lt;p&gt;In this blog, I will be explaining the task we were working on for the last 3-4 weeks. It will take you on a journey of optimizations from million graph traversals in building the database to just a few traversals in the end. Also, we will be covering the new architecture for the upcoming version of the Linked Commons and the reason behind the change.&lt;/p&gt;
&lt;h2 id="where-does-it-fit"&gt;Where does it fit?&lt;/h2&gt;&lt;p&gt;So far the Linked Commons was using a tiny subset of the data available in the CC Catalog. One of the primary targets of our team was to update the data. If you observe closely all tasks so far starting from adding "Graph Filtering Methods" to "Autocomplete Feature". These were actually bringing us closer towards this task. i.e. the much-awaited &lt;strong&gt;"Scale the Data of Linked Commons"&lt;/strong&gt;. We aim to add around &lt;strong&gt;235k nodes and 4.14 million links&lt;/strong&gt; into the Linked Commons project from around &lt;strong&gt;400 nodes and 500 links&lt;/strong&gt; in the current version. This drastic addition of new data is one of its kind, which makes this task very challenging and exciting.&lt;/p&gt;
&lt;h2 id="pilot"&gt;Pilot&lt;/h2&gt;&lt;p&gt;The raw CC Catalog data cannot be used directly in the Linked Commons. Our first task involves processing it, which includes removing isolated nodes, etc. You can read more about it in the data processing series &lt;a href="https://opensource.creativecommons.org/blog/entries/cc-datacatalog-data-processing/"&gt;blog&lt;/a&gt; written by my mentor Maria. After this, we need to build a database which stores the &lt;strong&gt;"distance list"&lt;/strong&gt; of all the nodes.&lt;/p&gt;
&lt;h3 id="what-is-distance-list"&gt;What is "distance list"?&lt;/h3&gt;&lt;div style="text-align: center; width: 90%; margin-left: 5%;"&gt;
    &lt;figure&gt;
        &lt;img src="distance-list.png" alt="Distance List" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;Distance list representation* of the node 'icij' part of a hypothetical graph&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;hr&gt;
&lt;p&gt;&lt;strong&gt;Distance List&lt;/strong&gt; is a method of graph representation. It is similar to &lt;a href="https://en.wikipedia.org/wiki/Adjacency_list"&gt;Adjacency List&lt;/a&gt; representation of graphs but instead of storing data of just immediate neighbouring nodes, "distance list" groups all vertices based on their distance from the root node and stores this grouped data for every vertex in the graph. In short, "distance list" is a more general form of the Adjacency List representation.&lt;/p&gt;
&lt;p&gt;To build this "distance list", we created a script for this, let’s name it &lt;strong&gt;build-dB-script.py&lt;/strong&gt;, which uses the &lt;a href="https://en.wikipedia.org/wiki/Breadth-first_search"&gt;Breadth-First Search(BFS)&lt;/a&gt; algorithm on every node to traverse the graph and gradually build this distance list. The filtering nodes feature of our web page connects to the server, which uses the aforementioned database and serves a smaller chunk of data.&lt;/p&gt;
&lt;h2 id="problem"&gt;Problem&lt;/h2&gt;&lt;p&gt;Now that we know where the &lt;em&gt;build-dB-script&lt;/em&gt; is used, let’s discuss the problems with it. The new graph data we are going to use is enormous and is in millions. A full traversal of a graph with million nodes, million times is very slow. Just to give some helpful numbers, the script was taking around 10 minutes to process a hundred nodes. Assuming the growth is linear(in the best case), it will take more than &lt;strong&gt;15 days&lt;/strong&gt; to complete the computations. &lt;strong&gt;It is scary, and thus, optimizations in the &lt;em&gt;build-dB-script&lt;/em&gt; are the need of the hour!!&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id="optimizations"&gt;Optimizations&lt;/h2&gt;&lt;p&gt;In this section, we will talk of the different versions of the build database script, starting from the brute force BFS method.&lt;/p&gt;
&lt;p&gt;The brute force BFS was the most simple and technically correct solution, but as the name suggests it was slow. In the next iteration, I stored the details of last n nodes, 10 to be precise and performed the same old BFS. It was faster but it had a logic error. Say, there is a link from a node to an already visited/traversed node. The script was not putting all the nodes which could have been explored from this path. After a few more leaps from Depth-first Search, to Breadth-first search, and other methods, eventually with the help of my mentors, we built a new approach - &lt;strong&gt;"Sequential dB Build"&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;To keep this blog short, I won’t be going too much into implementation details, but here are some of the critical points.&lt;/p&gt;
&lt;h3 id="key-points-of-the-sequential-db-build"&gt;Key points of the Sequential dB Build:&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;It was the fastest of all the predecessors and reduced the script timing significantly.&lt;/li&gt;
&lt;li&gt;In this approach, we aimed to build the all distance list of [1, 2, 3,... ., k-1] before building kth distance list.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Unfortunately, still, it was not enough for our current requirements. Just to give you some insights, the distance two list computation was taking around &lt;strong&gt;4 hours&lt;/strong&gt;, and &lt;strong&gt;distance three list&lt;/strong&gt; computation was taking &lt;strong&gt;20+ hours&lt;/strong&gt;. It shows that all these optimizations were not enough and were incapable of handling this big dataset.&lt;/p&gt;
&lt;h2 id="new-architecture"&gt;New Architecture&lt;/h2&gt;&lt;p&gt;As the optimizations in "build-dB-scripts" weren’t enough, we started looking to simplify the current architecture. In the end, we want to have a viable product which is scalable to this massive data. Although we are still not dropping the multi-distance filtering, we will continue our research on it and hopefully will have it in &lt;strong&gt;Linked Commons 3.0&lt;/strong&gt;. 😎&lt;/p&gt;
&lt;p&gt;For any node, it is more likely that any person would wish to know the immediate neighbours who are linking to some arbitrary node. Nodes at a distance greater than one exhibits very less information on the reach and connectivity of the root node. It was because of this we decided to change our current logic of having the distance list up to 10; instead, we reduced it to 1 and also stored the immediate incomming nodes list (Nodes which are at distance 1 in the &lt;a href="https://en.wikipedia.org/wiki/Transpose_graph"&gt;transpose graph&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;This small change in the design simplified a lot of things, and now the new graph build was taking around 2 minutes. By the time I am writing this blog we have upgraded our database from &lt;strong&gt;shelve to MongoDB&lt;/strong&gt; where the build time is further reduced. 🔥🔥&lt;/p&gt;
&lt;div style="text-align: center; width: 90%; margin-left: 5%;"&gt;
    &lt;figure&gt;
        &lt;img src="graph.png" alt="Light Theme" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;Graph showing neighbouring nodes. Incoming link are coloured with Turquoise and outgoing are coloured with Red.&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;&lt;p&gt;This task was really challenging and I learnt a lot. It was really mesmerizing to see the &lt;strong&gt;Linked Commons grow and evolve&lt;/strong&gt;. I hope you enjoyed reading this blog. You can follow the project development &lt;a href="https://github.com/creativecommons/cccatalog-dataviz/"&gt;here&lt;/a&gt;, and access the stable version of linked commons &lt;a href="http://dataviz.creativecommons.engineering/"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Feel free to report bugs and suggest features. It will help us improve this project. If you wish to join the our team, consider joining our &lt;a href="https://creativecommons.slack.com/channels/cc-dev-cc-catalog-viz"&gt;slack&lt;/a&gt; channel. Read more about our community teams &lt;a href="https://opensource.creativecommons.org/community/"&gt;here&lt;/a&gt;. See you in my next blog! 🚀&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;*&lt;em&gt;Linked Commons uses a more complex schema. The picture is just for illustration.&lt;/em&gt;&lt;/p&gt;
</content></entry><entry><title>CC Catalog: wrapping up GSoC20</title><link href="http://opensource.creativecommons.org/blog/entries/cc-catalog-wrapping-gsoc20/" rel="alternate"></link><updated>2020-08-25T00:00:00Z</updated><author><name>srinidhi</name></author><id>urn:uuid:ba947438-0d00-32ec-8ba7-acf6f5f15eb5</id><content type="html">&lt;p&gt;With the summer of code coming to an end, this blog post summarises the work done during the last three months. The project I have been working on is to add more provider API scripts to the CC Catalog. The CC Catalog project is responsible for collecting CC licensed images hosted across the web.&lt;/p&gt;
&lt;p&gt;The internship journey has been great , and I was glad to get the opportunity to understand more about the working of the data pipeline. My work during the internship mainly involved researching new API providers and checking if they meet the necessary conditions, then we decided on a strategy to crawl the API. The strategy varies according to different APIs:  some can be partitioned based on date, others have to be paginated . Script is written for the API according to the strategy. 
During the later phase of the internship, I had worked on the reingestion strategy for europeana and a script to merge Common Crawl tags and metadata to the corresponding image in the image table.&lt;/p&gt;
&lt;p&gt;Provider API implemented :&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Science Museum :  Science Museum collection has around 60,000 images and was initially crawled through Common Crawl and shifted to API based crawl. &lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://github.com/creativecommons/cccatalog/issues/302"&gt;Science Museum ticket&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Related PRs: &lt;a href="https://github.com/creativecommons/cccatalog/pull/400"&gt;Science Museum script&lt;/a&gt;, &lt;a href="https://github.com/creativecommons/cccatalog/pull/411"&gt;Science Museum workflow&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;ul&gt;
&lt;li&gt;Statens Museum : Statens Museum for Kunst is Denmark’s leading museum for artwork . This is a new integration and 39115 images have been collected.&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://github.com/creativecommons/cccatalog/issues/393"&gt;Statens Museum ticket&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Related PRs: &lt;a href="https://github.com/creativecommons/cccatalog/pull/428"&gt;Statens Museum implementation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;ul&gt;
&lt;li&gt;Museums Victoria : It was initially ingested from Common Crawl later shifted to API based crawl. It has around 140,000 images.&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://github.com/creativecommons/cccatalog/issues/291"&gt;Museums Victoria ticket&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Related PRs: &lt;a href="https://github.com/creativecommons/cccatalog/pull/447"&gt;Museums Victoria implementation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;ul&gt;
&lt;li&gt;NYPL : New York Public Library is a new integration , as of now it has around 1296 images.&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://github.com/creativecommons/cccatalog/issues/147"&gt;NYPL ticket&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Related PRs: &lt;a href="https://github.com/creativecommons/cccatalog/pull/462"&gt;NYPL implementation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;ul&gt;
&lt;li&gt;Brooklyn Museum : This was an existing integration , changes were made to follow the new &lt;code&gt;ImageStore&lt;/code&gt; and &lt;code&gt;DelayedRequestor&lt;/code&gt; class , it has 61503 images.&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://github.com/creativecommons/cccatalog/issues/348"&gt;Brooklyn Museum ticket&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Related PRs: &lt;a href="https://github.com/creativecommons/cccatalog/pull/355"&gt;Brooklyn Museum implementation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Iconfinder is a provider of icons that could not be integrated as the current strategy of ingestion is very slow and we need a better strategy.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Issue : &lt;a href="https://github.com/creativecommons/cccatalog/issues/396"&gt;Iconfinder ticket&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="europeana-reingestion-strategy"&gt;Europeana reingestion strategy&lt;/h2&gt;&lt;p&gt;Data collected from europeana was collected on a daily basis and there was a need to refresh it. The idea is that new data should be refreshed more frequently and as the data gets old, refreshing should become less frequent. While developing the strategy the API key limit and maximum collection expected is to be kept in mind. Considering these factors, a workflow was set up such that each day it crawls 59 days of data. 
The 59 days were split up into layers. The DAG crawls daily up to 1 week old data then it crawls monthly for data more than 1 week old and less than a year old data, anything older than a year is crawled every 3 months.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://github.com/creativecommons/cccatalog/issues/412"&gt;Europeana reingestion ticket&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Related PR: &lt;a href="https://github.com/creativecommons/cccatalog/pull/473"&gt;Europeana reingestion strategy&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;More details regarding the math of reingestion: &lt;a href="https://opensource.creativecommons.org/blog/entries/date-partitioned-data-reingestion/"&gt;Data reingestion&lt;/a&gt;&lt;/p&gt;
&lt;div style="text-align:center;"&gt;
    &lt;img src="dag_image_1.png" width="1000px"/&gt;
    &lt;img src="dag_image_2.png" width="1000px"/&gt;
    &lt;img src="dag_image_3.png" width="1000px"/&gt;
    &lt;p&gt;Europeana reingestion workflow&lt;/p&gt;
&lt;/div&gt;&lt;h2 id="merging-common-crawl-tags"&gt;Merging Common Crawl tags&lt;/h2&gt;&lt;p&gt;When a provider is shifted from Common Crawl to API based crawl, the new data from API doesn’t have tags and metadata that were generated using clarifai and hence there is need to associate the new data with the tags corresponding to that image from the Common Crawl data. A direct url match is not possible as the Common Crawl urls and API image url are different, so we try to match it on the number or identifier that is associated with the url.&lt;/p&gt;
&lt;p&gt;Currently the merging logic is applied to Science Museum, Museums Victoria and Met Museum .&lt;/p&gt;
&lt;p&gt;In Science Museum, API url in image table is like &lt;a href="https://coimages.sciencemuseumgroup.org.uk/images/240/862/large_BAB_S_1_02_0017.jpg"&gt;https://coimages.sciencemuseumgroup.org.uk/images/240/862/large_BAB_S_1_02_0017.jpg&lt;/a&gt; and CC url is like &lt;a href="https://s3-eu-west-1.amazonaws.com/smgco-images/images/369/541/medium_SMG00096855.jpg"&gt;https://s3-eu-west-1.amazonaws.com/smgco-images/images/369/541/medium_SMG00096855.jpg&lt;/a&gt; . So the idea is to reduce the url to the last identifier like number , so after the modification of the url by modify_urls function it looks like &lt;code&gt;gpj.1700_20_1_S_BAB_&lt;/code&gt; (API url) and &lt;code&gt;gpj.55869000GMS_&lt;/code&gt; (CC url) .
Similar logic has been applied to met museum and museum victoria.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://github.com/creativecommons/cccatalog/issues/468"&gt;https://github.com/creativecommons/cccatalog/issues/468&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Related PR: &lt;a href="https://github.com/creativecommons/cccatalog/pull/478"&gt;https://github.com/creativecommons/cccatalog/pull/478&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="acknowledgement"&gt;Acknowledgement&lt;/h2&gt;&lt;p&gt;I would like to thank my mentors Brent and Anna for their guidance throughout the internship.&lt;/p&gt;
</content></entry><entry><title>X5GON Using CC Catalog API for Image Results</title><link href="http://opensource.creativecommons.org/blog/entries/2020-08-x5gon-cc-catalog-api/" rel="alternate"></link><updated>2020-08-24T00:00:00Z</updated><author><name>annatuma</name></author><id>urn:uuid:ffcc37e0-31ad-3231-b583-73749555ba0b</id><content type="html">&lt;p&gt;A few months ago, the Open Education team at Creative Commons made an introduction between the folks working on X5GON and CC Search.&lt;/p&gt;
&lt;p&gt;Throughout a few conversations, we quickly discovered that there are many parallels to how we're approaching our work, and some important differences that would allow each of us to benefit from cooperation.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.x5gon.org/"&gt;X5GON&lt;/a&gt; is building an AI-driven platform, focused on delivery of open education resources (OER). At its core, it is building a catalog of OER, upon which other &lt;a href="https://www.x5gon.org/platforms/services/"&gt;services&lt;/a&gt; are based, such as analytics for personalized recommendations, and a discovery engine. By aggregating relevant content, curating it with the use of artificial intelligence and machine learning, and personalizing the experience to each learner, they're making OER more accessible and relevant.&lt;/p&gt;
&lt;p&gt;CC Search is not yet ready to ingest content types beyond images, but when we are able to do so, we plan to integrate via API with X5GON in order to serve OER that is made available in formats we will support in the future, starting with audio.&lt;/p&gt;
&lt;p&gt;The &lt;a href="https://discovery.x5gon.org/"&gt;X5GON Discovery search engine&lt;/a&gt; allows users to find OER in video, audio, and text formats - and now, with the integration of results powered by the CC Catalog API, which also powers CC Search, users can also find openly licensed images for relevant educational queries. This is a great resource for educators and learners from all over the world.&lt;/p&gt;
&lt;p&gt;Try it for yourself, or look at these results for making &lt;a href="https://discovery.x5gon.org/search?q=geometry&amp;amp;type=Image"&gt;geometry&lt;/a&gt; visual and fun!&lt;/p&gt;
</content></entry><entry><title>How to politely crawl and analyze 500 million images</title><link href="http://opensource.creativecommons.org/blog/entries/crawling-500-million/" rel="alternate"></link><updated>2020-08-17T00:00:00Z</updated><author><name>aldenpage</name></author><id>urn:uuid:c4bde8a8-0a5d-324a-b450-571f43e3af02</id><content type="html">&lt;h4 id="background"&gt;Background&lt;/h4&gt;&lt;p&gt;The goal of &lt;a href="https://search.creativecommons.org"&gt;CC Search&lt;/a&gt; is to index all of the Creative Commons works on the internet, starting with images. We have indexed over 500 million images, which we believe is roughly 36% of all CC licensed content on the internet by &lt;a href="https://creativecommons.org/2018/05/08/state-of-the-commons-2017/"&gt;our last count&lt;/a&gt;. To further enhance the usefulness of our search tool, we recently started crawling and analyzing images for improved search results. This article will discuss the process of taking a paper design for a large scale crawler, implementing it, and putting it in production, with a few idealized code snippets and diagrams along the way. The full source code can be viewed on &lt;a href="https://github.com/creativecommons/image-crawler"&gt;GitHub&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Originally, when we discovered an image and inserted it into CC Search, we didn't even bother downloading it; we stuck the URL in our database and embedded the image in our search results. This approach has a lot of problems:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;We don't know the dimensions or compression quality of images, which is useful both for relevance purposes (de-ranking low  quality images) and for filtering. For example, some users are only interested in high resolution images and would like to exclude content below a certain size.&lt;/li&gt;
&lt;li&gt;We can't run any type of computer vision analysis on any of the images, which could be useful for enriching search metadata through object recognition.&lt;/li&gt;
&lt;li&gt;Embedding third party content is fraught with problems. What if the other party's server goes down, the images disappear due to link rot, or their TLS certificates expire? Each of these situations results in broken images appearing in the search results or browser alerts about degraded security.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;We solved (3) by setting up a &lt;a href="https://github.com/willnorris/imageproxy"&gt;caching thumbnail proxy&lt;/a&gt; between images in the search results and their 3rd party origin, as well as some last-minute liveness checks to make sure that the image hasn't 404'd.&lt;/p&gt;
&lt;p&gt;(1) and (2), however, are not possible to solve without actually downloading the image and performing some analysis on the contents of the file. For us to reproduce the features that users take for granted in image search, we're going to need a fairly powerful crawling system.&lt;/p&gt;
&lt;p&gt;On the scale of several thousand images, it would be easy to cobble together a few scripts to spit out this information, but with half a billion images, there are a lot of hurdles to overcome.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;We want to crawl &lt;a href="https://en.wikipedia.org/wiki/Web_crawler#Politeness_policy"&gt;politely&lt;/a&gt;; however, the concentration and quantity of images means that we have to hit some sources with a high crawl rate in order to have any hope of finishing the crawl in a reasonable period of time. Our data sources range from non-profit museums with a single staff IT person to tech companies with their own data centers and thousands of employees; the crawl rate has to be tailored to download quickly from the big players but not overwhelm small sources. At the same time, we need to be sure that we are not overestimating any source's capacity and watch for signs that our crawler is straining the server.&lt;/li&gt;
&lt;li&gt;We need to keep the time to process each image as low as possible to make it feasible to finish the crawling and analysis task in a reasonable period of time. This means that the crawling and analysis tasks need to be distributed to multiple machines in parallel.&lt;/li&gt;
&lt;li&gt;A lot of metadata will be produced by this crawler. The step of integrating it with our internal systems needs to not block resizing tasks. That suggests that a message bus will be necessary to buffer messages before they are written into our data layer, where writes can be expensive.&lt;/li&gt;
&lt;li&gt;We want to have a basic idea of how the crawl is progressing in the form of summaries of error counts, status codes, and crawl rates for each source.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In summary, the challenge here isn't so much making a really fast crawler as much as it is tailoring the crawl speed to each source. At a minimum, we'll need to deal with concurrency and parallelism, provisioning and managing the life cycle of crawler infrastructure, pipelines for capturing output data, a way to monitor the progress of the crawl, a suite of tests to make sure the system behaves as expected, and a reliable way to enforce a "politeness" policy. That's not a trivial project, particularly for our tiny three person tech team (of which only one person is available to do all of the crawling work). Can't we just use an off-the-shelf open source crawler?&lt;/p&gt;
&lt;h4 id="what-about-existing-open-source-crawlers"&gt;What about existing open source crawlers?&lt;/h4&gt;&lt;p&gt;Any decent software engineer will consider existing options before diving into a project and reinventing the wheel. My assessment was that although there are a lot of open source crawling frameworks available, few of them focus on images, some are not actively maintained, and all would require extensive customization to meet the requirements of our crawl strategy. Further, many solutions are more complex than than our use case demands and would significantly expand our use of cloud infrastructure, resulting in higher expenses and more operational headaches. I experimented with Apache Nutch, Scrapy Cluster, and Frontera; none of the existing options looked quite right for our use case.&lt;/p&gt;
&lt;p&gt;As a reminder, we want to eventually crawl every single Creative Commons work on the internet. Effective crawling is central to the capabilities that our search engine is able to provide. In addition to being central to achieving high quality image search, crawling could also be useful for discovering new Creative Commons content of any type on any website. In my view, that's a strong argument for spending some time designing a custom crawling solution where we have complete end-to-end control of the process, as long as the feature set is limited in scope. In the next section, we'll assess the effort required to build a crawler from the ground up.&lt;/p&gt;
&lt;h4 id="designing-the-crawler"&gt;Designing the crawler&lt;/h4&gt;&lt;p&gt;We know we're not going to be able to crawl 500 million images with one virtual machine and a single IP address, so it is obvious from the start that we are going to need a way to distribute the crawling and analysis tasks over multiple machines. A basic queue-worker architecture will do the job here; when we want to crawl an image, we can dispatch the URL to an inbound images queue, and a worker eventually pops that task out and processes it. Kafka will handle all of the hard work of partitioning and distributing the tasks between workers.&lt;/p&gt;
&lt;p&gt;The worker processes do the actual analysis of the images, which essentially entails downloading the image, extracting interesting properties, and sticking the resulting metadata back into a Kafka topic for later downstream processing. The worker will also have to include some instrumentation for conforming to rate limits and error reporting.&lt;/p&gt;
&lt;p&gt;We also know that we will need to share some information about crawl progress between worker processes, such as whether we've exceeded our prescribed rate limit for a website, the number of times we've seen a status code in the last minute, how many images we've processed so far, and so on. Since we're only interested in sharing application state and aggregate statistics, a lightweight key/value store like Redis seems like a good fit.&lt;/p&gt;
&lt;p&gt;Finally, we need a supervising process that centrally controls the crawl. This key governing process will be responsible for making sure our crawler workers are behaving properly by moderating crawl rates for each source, taking action in the face of errors, and reporting statistics to the operators of the crawler. We'll call this process the crawl monitor.&lt;/p&gt;
&lt;p&gt;Here's a rough sketch of how things will work:&lt;/p&gt;
&lt;p&gt;&lt;img src="/blog/entries/crawling-500-million/image_crawler_simplified.png" alt="Diagram"&gt;&lt;/p&gt;
&lt;p&gt;At a high level, the problem of building a fast crawler seems solvable for our team, even on the scale of several hundred million images. If we can sustain a crawl and analysis rate of 200 images per second, we could crawl all 500 million images in about a month.&lt;/p&gt;
&lt;p&gt;In the next section, we'll examine some of the key components that make up the crawler.&lt;/p&gt;
&lt;h4 id="detailed-breakdown"&gt;Detailed breakdown&lt;/h4&gt;&lt;h5 id="concurrency-with-asyncio"&gt;Concurrency with &lt;code&gt;asyncio&lt;/code&gt;&lt;/h5&gt;&lt;p&gt;Crawling is a massively IO bound task. The workers need to maintain lots of simultaneous open connections with internal systems like Kafka and Redis as well as 3rd party websites holding the target images. Once we have the image in memory, performing our actual analysis task is easy and cheap. For these reasons, an asynchronous approach seems more attractive than using multiple threads of execution. Even if our image processing task grows in complexity and becomes CPU bound, we can get the best of both worlds by offloading heavyweight tasks to a process pool. See "&lt;a href="https://docs.python.org/3/library/asyncio-dev.html#running-blocking-code"&gt;Running Blocking Code&lt;/a&gt;" in the &lt;code&gt;asyncio&lt;/code&gt; docs for more details.&lt;/p&gt;
&lt;p&gt;Another reason that an asynchronous approach may be desirable is that we have several interlocking components which need to react to events in real-time: our crawl monitoring process needs to simultaneously control the rate limiting process and also interrupt crawling if errors are detected, while our worker processes need to consume crawl events, process images, upload thumbnails, and produce events documenting the metadata of each image. Coordinating all of these components through inter-process communication could be difficult, but breaking up tasks into small pieces and yielding to the event loop is comparatively easy.&lt;/p&gt;
&lt;h5 id="the-resize-task"&gt;The resize task&lt;/h5&gt;&lt;p&gt;This is the most vital part of our crawling system: the part that actually does the work of fetching and processing an image. As established previously, we need to execute this task concurrently, so everything needs to be defined with &lt;code&gt;async&lt;/code&gt;/&lt;code&gt;await&lt;/code&gt; syntax to allow the event loop to multitask. The actual task itself is otherwise straightforward.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Download the remote image and load it into memory.&lt;/li&gt;
&lt;li&gt;Extract the resolution and compression quality.&lt;/li&gt;
&lt;li&gt;Thumbnail the image for later computer vision analysis and upload it to S3.&lt;/li&gt;
&lt;li&gt;Write the information we've discovered to a Kafka topic.&lt;/li&gt;
&lt;li&gt;Report success/errors to Redis in aggregate.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;See &lt;a href="https://github.com/creativecommons/image-crawler/blob/master/worker/image.py"&gt;image.py&lt;/a&gt; for the nitty-gritty details.&lt;/p&gt;
&lt;h4 id="rate-limiting-with-token-buckets-and-error-circuit-breakers"&gt;Rate limiting with token buckets and error circuit breakers&lt;/h4&gt;&lt;h5 id="how-do-we-determine-the-rate-limit"&gt;How do we determine the rate limit?&lt;/h5&gt;&lt;p&gt;Often times, when designing highly concurrent software, the goal is to maximize the throughput and push servers to their absolute limit. The opposite is true with a web crawler, particularly when you are operating a non-profit organization completely reliant on the goodwill of others to exist. We want to be as certain as reasonably possible that we aren't going to knock a resource off of the internet with an accidental &lt;a href="https://en.wikipedia.org/wiki/Denial-of-service_attack"&gt;DDoS&lt;/a&gt;. At the same time, we need to crawl as quickly as possible against sources with adequate resources to withstand a heavy crawl, or else we'll never finish. How can we match our crawl rate to a site's capabilities?&lt;/p&gt;
&lt;p&gt;Originally, my plan was to determine this through an adaptive rate limiting strategy, where we would start with a low rate limit and use a hill climbing algorithm to determine the optimal rate. We could track metrics like &lt;a href="https://en.wikipedia.org/wiki/Time_to_first_byte"&gt;time to first byte&lt;/a&gt; (TTFB) and bandwidth speed to determine the exact moment that we have started to strain upstream servers. However, there are a lot of drawbacks here:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;It may not be correct to assume that performance will steadily degrade instead of failing all at once.&lt;/li&gt;
&lt;li&gt;We can't detect whether we are the cause of a performance issue or if the host is simply experiencing server trouble due to configuration errors or high traffic. We could get stuck at a suboptimal rate limit due to normal fluctuations in traffic.&lt;/li&gt;
&lt;li&gt;Recording TTFB in Python is difficult because it requires low level access to connection data. We might have to write an extension to &lt;code&gt;aiohttp&lt;/code&gt; to get it.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Eventually I decided that this is too much hassle. Can we get the job done with a simpler strategy?&lt;/p&gt;
&lt;p&gt;It turns out that the size of a website is typically correlated with infrastructure capabilities. The reasoning behind this is that if you are capable of hosting 450MM images, you are probably able to handle at least a couple hundred requests per second for serving traffic. In our case, we already know how many images a source has, so it's easy for us to peg our rate limit between a low minimum for small websites and a reasonable maximum for large websites, and then interpolate the rate limit for everything in between.&lt;/p&gt;
&lt;p&gt;Of course, it's important to note that this is only a rough heuristic that we use to make a reasonable guess about what a website can handle. We have to allow the possibility that we set our rate limit too aggressively in spite of our precautions.&lt;/p&gt;
&lt;h5 id="backing-off-with-circuit-breakers"&gt;Backing off with circuit breakers&lt;/h5&gt;&lt;p&gt;If our heuristic fails to correctly approximate the bandwidth capabilities of a site, we are going to start encountering problems. For one, we might exceed the server-side rate limit, which means we will see &lt;code&gt;429 Rate Limit Exceeded&lt;/code&gt; and &lt;code&gt;403 Forbidden&lt;/code&gt; errors instead of the images we're trying to crawl. Worse yet, the upstream source might continue to happily serve requests while we suck up all of their traffic capacity, resulting in other users being unable to view the images. Clearly, in either scenario, we need to either reduce our crawl rate or even give up crawling the source entirely if it appears that we are impacting their uptime.&lt;/p&gt;
&lt;p&gt;To handle these situations, we have two tools in our toolbox: a sliding window recording the status code of every request made we've made to each domain in the last 60 seconds, and a list of the last 50 statuses for each website. If the number of errors in our one minute window exceed 10%, something is wrong; we should wait a minute before trying again. If we have encountered many errors in a row, however, that suggests that we're having trouble with a particular site, so we ought to give up crawling the source and raise an alert.&lt;/p&gt;
&lt;p&gt;Workers can keep track of this information in sorted sets in Redis. For the sliding error window, we'll sort each request by its timestamp, which will make it easy and cheap for us to expire status codes beyond the sliding window interval. Maintaining a list of the last N response codes is even easier; we just stick the status code in a list associated with the source.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;StatsManager&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="fm"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;redis&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;
        &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;known_sources&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="nd"&gt;@staticmethod&lt;/span&gt;
    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_record_window_samples&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pipe&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot; Insert a status into all sliding windows. &amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
        &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="c1"&gt;# Time-based sliding windows&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;stat_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;WINDOW_PAIRS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;stat_key&lt;/span&gt;&lt;span class="si"&gt;}{&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
            &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;pipe&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;zadd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s1"&gt;:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="c1"&gt;# Delete events from outside the window&lt;/span&gt;
            &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;pipe&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;zremrangebyscore&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;-inf&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="c1"&gt;# &amp;quot;Last n requests&amp;quot; window&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;pipe&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rpush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;LAST_50_REQUESTS&lt;/span&gt;&lt;span class="si"&gt;}{&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;pipe&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ltrim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;LAST_50_REQUESTS&lt;/span&gt;&lt;span class="si"&gt;}{&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;Collecting status codes in aggregate&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Meanwhile, the crawl monitor process can keep tabs on the contents of each error threshold.&lt;/p&gt;
&lt;p&gt;When more than 10% of the requests made to a source in the last minute are errors, we'll set a halt condition in Redis and stop replenishing rate limit tokens (more on that below).&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;monotonic&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;one_minute_window&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;zrangebyscore&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;one_minute_window_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;-inf&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;errors&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="n"&gt;success&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;one_minute_window&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;EXPECTED_STATUSES&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;errors&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;successful&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="n"&gt;tolerance&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ERROR_TOLERANCE_PERCENT&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;successful&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;successful&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;tolerance&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sadd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TEMP_HALTED_SET&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;Detecting elevated crawl errors for a source&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;For detecting "serious" errors, where we've seen 50 failed requests in a row, we'll set a permanent halt condition. Someone will have to manually troubleshoot the situation and switch the crawler back on for that source.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;last_50_statuses_key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;statuslast50req:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
&lt;span class="n"&gt;last_50_statuses&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lrange&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;last_50_statuses_key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;last_50_statuses&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;_every_request_failed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;last_50_statuses&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sadd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;HALTED_SET&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;Detecting persistent crawl errors&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;In practice, keeping a sliding window for tracking error thresholds and setting reasonable minimum and maximum crawl rates has worked well enough that the circuit breaker has never been tripped.&lt;/p&gt;
&lt;h5 id="enforcing-rate-limits-with-token-buckets"&gt;Enforcing rate limits with token buckets&lt;/h5&gt;&lt;p&gt;It's one thing to set a policy for crawling; it's another thing entirely to actually enforce it. How can we coordinate our multiple crawling processes to prevent them from overstepping our rate limit?&lt;/p&gt;
&lt;p&gt;The answer is to implement a distributed token bucket system. The idea behind this is that each crawler has to obtain a token from Redis before making a request. Every second, the crawl monitor sets a variable containing the number of requests that can be made against a source. Each crawler process decrements the counter before making a request. If the decremented result is above zero, the worker is cleared to crawl. Otherwise, the rate limit has been reached and we should wait until a token has been obtained.&lt;/p&gt;
&lt;p&gt;The beauty of token buckets is their simplicity, performance, and resilience against failure. If our crawler monitor process dies, crawling halts completely; making a request is not possible without first acquiring a token. This is a much better alternative to the guard rails completely disappearing with the crawl monitor and allowing unbounded crawling. Further, since decrementing a counter and retrieving the result is an atomic operation in Redis, there's no risk of race conditions and therefore no need for locking. This is a huge boon for performance, as the overhead of coordinating and blocking on every single request would rapidly bog down our crawling system.&lt;/p&gt;
&lt;p&gt;To ensure that all crawling is performed at the correct speed, I wrapped &lt;code&gt;aiohttp.ClientSession&lt;/code&gt; with a rate limited version of the class.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;RateLimitedClientSession&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="fm"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;aioclient&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;aioclient&lt;/span&gt;
        &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;redis&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;

    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_get_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;token_key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;CURRTOKEN_PREFIX&lt;/span&gt;&lt;span class="si"&gt;}{&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
        &lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;decr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token_key&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;token_acquired&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="c1"&gt;# Out of tokens&lt;/span&gt;
            &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;token_acquired&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;token_acquired&lt;/span&gt;

    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;token_acquired&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;
        &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;token_acquired&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;token_acquired&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_get_token&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Meanwhile, the crawl monitor process is filling up each bucket every second.&lt;/p&gt;
&lt;h5 id="scheduling-tasks-somewhat-intelligently"&gt;Scheduling tasks (somewhat) intelligently&lt;/h5&gt;&lt;p&gt;The final gotcha in the design of our crawler is that we want to crawl every single website at the same time at its prescribed rate limit. That sounds almost tautological, like something that we should be able to take for granted after implementing all of this logic for preventing our crawler from working too quickly, but it turns out our crawler's processing capacity itself is a limited and contentious resource. We can only schedule so many tasks simultaneously on each worker, and we need to ensure that tasks from a single website aren't starving other sources of crawl capacity.&lt;/p&gt;
&lt;p&gt;For instance, imagine that each worker is able to handle 5000 simultaneous crawling tasks, and every one of those tasks is tied to a tiny website with a very low rate limit. That means that our entire worker, which is capable of handling hundreds of crawl and analysis jobs per second, is stuck making one request per second until some faster tasks appear in the queue.&lt;/p&gt;
&lt;p&gt;In other words, we need to make sure that each worker process isn't jamming itself up with a single source. We have a &lt;a href="https://en.wikipedia.org/wiki/Scheduling_(computing%29"&gt;scheduling problem&lt;/a&gt;. We've naively implemented first-come-first-serve and need to switch to a different scheduling strategy.&lt;/p&gt;
&lt;p&gt;There are innumerable ways to address scheduling problems. Since there are only a few dozen sources in our system, we can get away with using a stupid scheduling algorithm: give each source equal capacity in every worker. In other words, if there are 5000 tasks to distribute and 30 sources, we can allocate 166 simultaneous tasks to each source per worker. That's plenty for our purposes. There are obvious drawbacks of this approach in that eventually there will be so many sources that we start starving high rate limit sources of work. We'll cross that bridge when we come to it; it's better to use the simplest possible approach we can get away with instead of spending all of our time on solving hypothetical future problems.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_schedule&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;task_schedule&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;raw_sources&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;smembers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;inbound_sources&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;sources&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;utf-8&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;raw_sources&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="n"&gt;num_sources&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="c1"&gt;# A source never gets more than 1/4th of the worker&amp;#39;s capacity. This&lt;/span&gt;
        &lt;span class="c1"&gt;# helps prevent starvation of lower rate limit requests and ensures&lt;/span&gt;
        &lt;span class="c1"&gt;# that the first few sources to be discovered don&amp;#39;t get all of the&lt;/span&gt;
        &lt;span class="c1"&gt;# initial task slots.&lt;/span&gt;
        &lt;span class="n"&gt;max_share&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;settings&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MAX_TASKS&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;
        &lt;span class="n"&gt;share&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;settings&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MAX_TASKS&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;num_sources&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;max_share&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;to_schedule&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;num_unfinished&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_get_unfinished_tasks&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;task_schedule&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;num_to_schedule&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;share&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;num_unfinished&lt;/span&gt;
            &lt;span class="n"&gt;consumer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_get_consumer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;source_msgs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_consume_n&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;consumer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;num_to_schedule&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;to_schedule&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;source_msgs&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;to_schedule&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;Scheduling tasks for every source&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The one implementation detail to deal with here is that our workers can't draw from a single inbound images queue anymore; we need to partition each source into its own queue so we can pull tasks from each source when we need it. This partitioning process can be handled transparently by the crawl monitor.&lt;/p&gt;
&lt;p&gt;&lt;img src="/blog/entries/crawling-500-million/image_crawler.png" alt="A more complete diagram"&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;A more complete diagram showing the system with a queue for each source&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;
&lt;h5 id="designing-for-testability"&gt;Designing for testability&lt;/h5&gt;&lt;p&gt;It's quite difficult to test IO-heavy systems because of their need to interact with lots of external dependencies. Often times it is necessary to write complex integration tests or run manual tests to be certain that key functionality works as expected. This is no good because integration tests are much more expensive to maintain and take far longer to execute. We certainly wouldn't go to production without running a smoke test to verify correctness in real-world conditions, but it's still critical to have unit tests in place for catching bugs quickly during the development process.&lt;/p&gt;
&lt;p&gt;The solution to this problem is to use dependency injection, which is a fancy way of saying that we never do IO directly from within our application. Instead, we delegate IO to external objects that can be passed in at run-time. This makes it easy to pass in fake objects that approximate real world behavior without real world consequences.&lt;/p&gt;
&lt;p&gt;For example, the crawl monitor usually has to talk to our CC Search API (for assessing source size), Redis, and Kafka to do its job of regulating the crawl; instead of setting up a brittle and complicated integration test with all of those dependencies, we just instantiate some mock objects and pass them in. Now we can easily test individual components such as the error circuit breaker.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;Testing our crawl monitor's circuit breaking functionality with mock dependencies&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="nd"&gt;@pytest&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fixture&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;source_fixture&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot; Mocks the /v1/sources endpoint response. &amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="s2"&gt;&amp;quot;source_name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;example&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;&amp;quot;image_count&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;&amp;quot;display_name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Example&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;&amp;quot;source_url&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;example.com&amp;quot;&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="s2"&gt;&amp;quot;source_name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;another&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;&amp;quot;image_count&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1000000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;&amp;quot;display_name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;Another&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;&amp;quot;source_url&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;whatever&amp;quot;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;create_mock_monitor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;FakeAioResponse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;FakeAioSession&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;redis&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;FakeRedis&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;regulator_task&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;create_task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rate_limit_regulator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;regulator_task&lt;/span&gt;


&lt;span class="nd"&gt;@pytest&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mark&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;asyncio&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;test_error_circuit_breaker&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source_fixture&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;sources&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;source_fixture&lt;/span&gt;
    &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;monitor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;create_mock_monitor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sources&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;statuslast50req:example&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;500&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;
    &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;statuslast50req:another&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;200&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;run_monitor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;monitor_task&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;example&amp;#39;&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;halted&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;another&amp;#39;&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;store&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;halted&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;The main drawback of dependency injection is that initializing your objects will take some more ceremony. See the &lt;a href="https://github.com/creativecommons/image-crawler/blob/00b59aba9a15faccf203a53d73a98e8c06cb69e8/worker/scheduler.py#L162"&gt;initialization of the crawl scheduler&lt;/a&gt; for an example of wiring up an object with a lot of dependencies. You might also find that constructors and other functions with a lot of dependencies will have a lot of arguments if care isn't taken to bundle external dependencies together. In my opinion, the price of a few extra lines of initialization code is well worth the benefits gained from testability and modularity.&lt;/p&gt;
&lt;h4 id="smoke-testing"&gt;Smoke testing&lt;/h4&gt;&lt;p&gt;Even with our unit test coverage, we still need to do some basic small-scale manual tests to make sure our assumptions hold up in the real world. We'll need to write &lt;a href="https://www.terraform.io/"&gt;Terraform&lt;/a&gt; modules that provision a working version of the real system. Sadly, our Terraform infrastructure repository is private for now, but here's a taste of what the infra code looks like.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kr"&gt;module&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;&amp;quot;image-crawler&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;source&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;../../modules/services/image-crawler&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;prod&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;docker_tag&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0.25.0&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;aws_access_key_id&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.aws_access_key_id}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;aws_secret_access_key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.aws_secret_access_key}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;zookeeper_endpoint&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${module.kafka.zookeeper_brokers}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;kafka_brokers&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${module.kafka.kafka_brokers}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;worker_instance_type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;m5.large&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;worker_count&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;Initialization of crawler Terraform module in our production environment&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="kr"&gt;resource&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;&amp;quot;aws_instance&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;&amp;quot;crawler-workers&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;ami&lt;/span&gt;&lt;span class="w"&gt;                     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.ami}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;instance_type&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.worker_instance_type}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;user_data&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${data.template_file.worker_init.rendered}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;subnet_id&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${element(data.aws_subnet_ids.subnets.ids, 0)}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;vpc_security_group_ids&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${aws_security_group.image-crawler-sg.id}&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;count&lt;/span&gt;&lt;span class="w"&gt;                   &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.worker_count}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nb"&gt;tags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;image-crawler-worker-${var.environment}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.environment}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc:environment&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.environment == &amp;quot;dev&amp;quot; ? &amp;quot;staging&amp;quot; : &amp;quot;production&amp;quot;}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc:product&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cccatalog-api&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc:purpose&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Image crawler worker&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc:team&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc-search&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;

&lt;span class="kr"&gt;resource&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;&amp;quot;aws_instance&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;&amp;quot;crawler-monitor&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;ami&lt;/span&gt;&lt;span class="w"&gt;                     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.ami}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;instance_type&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;c5.large&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;user_data&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${data.template_file.monitor_init.rendered}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;subnet_id&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${element(data.aws_subnet_ids.subnets.ids, 0)}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;vpc_security_group_ids&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${aws_security_group.image-crawler-sg.id}&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nb"&gt;tags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;image-crawler-monitor-${var.environment}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.environment}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc:environment&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;${var.environment == &amp;quot;dev&amp;quot; ? &amp;quot;staging&amp;quot; : &amp;quot;production&amp;quot;}&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc:product&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cccatalog-api&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc:purpose&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Image crawler monitor&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc:team&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cc-search&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;&lt;em&gt;&lt;center&gt;An excerpt of the crawler module definition&lt;/center&gt;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;One &lt;code&gt;terraform plan&lt;/code&gt; and &lt;code&gt;terraform apply&lt;/code&gt; cycle later, we're ready to feed a few million test URLs to the inbound image queue and see what happens. By my recollection, this uncovered many glaring issues:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Basic network security configuration problems preventing communication between key components&lt;/li&gt;
&lt;li&gt;The need for our scheduling algorithm to be overhauled (already discussed)&lt;/li&gt;
&lt;li&gt;Workers exceeding Redis maximum connection limit&lt;/li&gt;
&lt;li&gt;Workers crashing due to hitting open file limit due to huge number of concurrent connections&lt;/li&gt;
&lt;li&gt;Probably a half dozen other problems&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;After fixing all of those issues and performing a larger smoke test, we're ready to start crawling on a large scale.&lt;/p&gt;
&lt;h5 id="monitoring-the-crawl"&gt;Monitoring the crawl&lt;/h5&gt;&lt;p&gt;Unfortunately, we can't just kick back and relax while the crawler does its thing for a few weeks. We need some transparency about what the crawler is doing so we can be alerted when something breaks.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;How fast are we crawling each website? What's our target rate limit?&lt;/li&gt;
&lt;li&gt;How many errors have occurred? How many images have we successfully processed?&lt;/li&gt;
&lt;li&gt;Are we crawling right now, or are we finished?&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;It would be nice to build a reporting dashboard for this, but in the interest of time, we'll dump a giant JSON blob to &lt;code&gt;STDOUT&lt;/code&gt; every 5 seconds and call it a day. When we want to check on crawl progress, we &lt;code&gt;ssh&lt;/code&gt; into the crawl monitoring virtual machine and &lt;code&gt;tail&lt;/code&gt; the logs (we could also use our Graylog instance if we're feeling lazy). Fortunately, JSON is both trivially human and machine readable, so we can build a more sophisticated monitoring system later by parsing the logs.&lt;/p&gt;
&lt;p&gt;Here's an example log line from one of our smoke tests, indicating that we've crawled 13,224 images successfully and nothing else is happening.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;event&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;monitoring_update&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;time&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2020-04-17T20:22:56.837232&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;general&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;global_max_rps&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;193.418869804698&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;error_rps&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;processing_rate&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;success_rps&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;circuit_breaker_tripped&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;num_resized&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;13224&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;resize_errors&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;split_rate&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;specific&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;flickr&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;successful&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;13188&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;last_50_statuses&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;200&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;rate_limit&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;178.375147633876&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;error&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;animaldiversity&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;last_50_statuses&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;200&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;successful&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;error&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;rate_limit&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.206215440554406&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;phylopic&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;rate_limit&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0.2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;error&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;successful&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;last_50_statuses&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;200&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Now that we can see what the crawler is up to, we can schedule the larger crawl and start collecting production quality data.&lt;/p&gt;
&lt;h4 id="takeaways"&gt;Takeaways&lt;/h4&gt;&lt;p&gt;The result here is that we have a lightweight, modular, highly concurrent, and polite distributed image crawler with only a handful of lines of code.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;alden:~/code/image_crawler$ cloc .
      &lt;span class="m"&gt;48&lt;/span&gt; text files.
      &lt;span class="m"&gt;43&lt;/span&gt; unique files.                              
      &lt;span class="m"&gt;25&lt;/span&gt; files ignored.

github.com/AlDanial/cloc v &lt;span class="m"&gt;1&lt;/span&gt;.81  &lt;span class="nv"&gt;T&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.02 s &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1667&lt;/span&gt;.4 files/s, &lt;span class="m"&gt;130887&lt;/span&gt;.8 lines/s&lt;span class="o"&gt;)&lt;/span&gt;
------------------------------------------------------------------------------
Language                     files          blank        comment           code
------------------------------------------------------------------------------
Python                          &lt;span class="m"&gt;16&lt;/span&gt;            &lt;span class="m"&gt;244&lt;/span&gt;            &lt;span class="m"&gt;242&lt;/span&gt;           &lt;span class="m"&gt;1324&lt;/span&gt;
Markdown                         &lt;span class="m"&gt;5&lt;/span&gt;             &lt;span class="m"&gt;79&lt;/span&gt;              &lt;span class="m"&gt;0&lt;/span&gt;            &lt;span class="m"&gt;219&lt;/span&gt;
YAML                             &lt;span class="m"&gt;3&lt;/span&gt;              &lt;span class="m"&gt;2&lt;/span&gt;              &lt;span class="m"&gt;4&lt;/span&gt;             &lt;span class="m"&gt;61&lt;/span&gt;
XML                              &lt;span class="m"&gt;3&lt;/span&gt;              &lt;span class="m"&gt;0&lt;/span&gt;              &lt;span class="m"&gt;0&lt;/span&gt;             &lt;span class="m"&gt;18&lt;/span&gt;
Bourne Shell                     &lt;span class="m"&gt;1&lt;/span&gt;              &lt;span class="m"&gt;0&lt;/span&gt;              &lt;span class="m"&gt;1&lt;/span&gt;              &lt;span class="m"&gt;4&lt;/span&gt;
------------------------------------------------------------------------------
SUM:                            &lt;span class="m"&gt;28&lt;/span&gt;            &lt;span class="m"&gt;325&lt;/span&gt;            &lt;span class="m"&gt;247&lt;/span&gt;           &lt;span class="m"&gt;1626&lt;/span&gt;
------------------------------------------------------------------------------

alden:~/code/image_crawler$ tree .
.
├── architecture.png
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── crawl_monitor
│   ├── __init__.py
│   ├── monitor.py
│   ├── rate_limit.py
│   ├── README.md
│   ├── settings.py
│   ├── source_splitter.py
│   ├── structured_logging.py
│   └── tsv_producer.py
├── docker-compose.yml
├── Dockerfile-monitor
├── Dockerfile-worker
├── __init__.py
├── LICENSE
├── Pipfile
├── Pipfile.lock
├── publish_release.sh
├── README.md
├── &lt;span class="nb"&gt;test&lt;/span&gt;
│   ├── corrupt.jpg
│   ├── __init__.py
│   ├── mocks.py
│   ├── test_image.jpg
│   ├── test_monitor.py
│   └── test_worker.py
└── worker
    ├── image.py
    ├── __init__.py
    ├── message.py
    ├── rate_limit.py
    ├── scheduler.py
    ├── settings.py
    ├── stats_reporting.py
    └── util.py

&lt;span class="m"&gt;3&lt;/span&gt; directories, &lt;span class="m"&gt;34&lt;/span&gt; files
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;We now have a lot of useful information about images that we were lacking before. The next step is to take this metadata and integrate it into our search engine, as well as perform deeper analysis of images using computer vision.&lt;/p&gt;
</content></entry><entry><title>Say Hello To Our Community Team</title><link href="http://opensource.creativecommons.org/blog/entries/say_hello_to_ct/" rel="alternate"></link><updated>2020-08-14T00:00:00Z</updated><author><name>dhruvkb</name></author><id>urn:uuid:ddeb63da-7771-357d-8299-ad51defeaf4a</id><content type="html">&lt;p&gt;Creative Commons is committed to open-source software. We have over two dozen
projects, spanning three times as many repositories on GitHub, each with its
small, but extremely enthusiastic, subcommunity. With only a few full-time
employees working on these projects, it is vital that we enable members from the
community to take increased responsibility in developing and maintaining them,
and growing the community of which they are a part.&lt;/p&gt;
&lt;p&gt;With that goal in mind, we've launched our Community Team initiative.&lt;/p&gt;
&lt;h3 id="what-is-the-community-team"&gt;What is the Community Team?&lt;/h3&gt;&lt;p&gt;Communities that grow organically around open source projects tend to be a bit
disorganised and the frequency of contributions and degree of involvement tends
to vary from member to member. Our goal is to identify contributors who are
actively involved within their communities and give them increased permissions
over the codebase and access to more information channels and tools in an effort
to empower them to participate more fully in the project.&lt;/p&gt;
&lt;p&gt;This is not restricted to code though. We're also looking for people who work
with the community on other aspects of the projects, such as design,
documentation, evangelism, and onboarding to name a few.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The Community Team establishes a framework for formalising the level of
involvement, which is a spectrum, into discrete level or 'roles'.&lt;/li&gt;
&lt;li&gt;Each role is mapped to a set of responsibilities that a member holding the
role is encouraged to take up.&lt;/li&gt;
&lt;li&gt;Each role also entrusts the members holding it to certain privileges, accesses
and permissions, to help them execute these responsibilities.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Roles also progressively include members in our roadmaps and planning meetings
to ensure that the community is aligned with our long-term goals.&lt;/p&gt;
&lt;h3 id="what-s-in-it-for-me"&gt;What's in it for me?&lt;/h3&gt;&lt;p&gt;The Community Team is not just a one-sided deal. Your membership in the
Community Team is just as beneficial for the you as it is for us. While there is
a &lt;a href="/community/community-team/#benefits-of-joining-the-community-team"&gt;laundry list of benefits&lt;/a&gt; that you're entitled to, I'll just
mention some notable ones here.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;You gain real-world practical experience of working on open-source projects.&lt;/li&gt;
&lt;li&gt;You gain both soft-skills and technical-skills by interacting with other
developers from both the community as well as CC staff.&lt;/li&gt;
&lt;li&gt;Since we've already seen the quality of your work and involvement with the
community, you get priority in internship applications*.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Oh and, lest I forget, you'll receive CC swag!&lt;/p&gt;
&lt;p&gt;&lt;blockquote class="twitter-tweet" data-align="center"&gt;
  &lt;p lang="en" dir="ltr"&gt;
    Thanks for the goodies!!
    &lt;a href="https://twitter.com/creativecommons?ref_src=twsrc%5Etfw"&gt;@creativecommons&lt;/a&gt;
    😀
    &lt;a href="https://twitter.com/hashtag/OpenSource?src=hash&amp;amp;ref_src=twsrc%5Etfw"&gt;#OpenSource&lt;/a&gt;
    &lt;a href="https://twitter.com/hashtag/creativecommons?src=hash&amp;amp;ref_src=twsrc%5Etfw"&gt;#creativecommons&lt;/a&gt;
    &lt;a href="https://twitter.com/hashtag/GSoC?src=hash&amp;amp;ref_src=twsrc%5Etfw"&gt;#GSoC&lt;/a&gt;
    &lt;a href="https://t.co/DFvpXCs8uu"&gt;pic.twitter.com/DFvpXCs8uu&lt;/a&gt;
  &lt;/p&gt;
  &amp;mdash;
  Mayank Nader (@MayankNader)
  &lt;a href="https://twitter.com/MayankNader/status/1137995920866390016?ref_src=twsrc%5Etfw"&gt;June 10, 2019&lt;/a&gt;
&lt;/blockquote&gt;&lt;/p&gt;
&lt;script async src="https://platform.twitter.com/widgets.js" charset="utf-8"&gt;&lt;/script&gt;&lt;h3 id="what-are-these-roles"&gt;What are these 'roles'?&lt;/h3&gt;&lt;p&gt;If you've reached this point, I assume you see the potential of the Community
Team. Let's see where you'd fit in them.&lt;/p&gt;
&lt;p&gt;We have two kinds of roles, code-oriented &lt;a href="/community/community-team/project-roles/"&gt;Project roles&lt;/a&gt;, that
give you responsibilities and permissions related to one CC project, and
non-code-oriented &lt;a href="/community/community-team/community-building-roles/"&gt;Community Building roles&lt;/a&gt;, that give
you responsibilities and permissions related to improving the community of all
CC projects as a whole.&lt;/p&gt;
&lt;p&gt;Each type has a few levels but that I'll just link them for you to read on your
own. While your eligibility for any role depends on how involved you have been
in the past, the role you choose reflects how involved you would like to be in
the future.&lt;/p&gt;
&lt;p&gt;Start by asking yourself a simple question, "Do I code?"&lt;/p&gt;
&lt;h4 id="sure-i-can-code..""&gt;"Sure, I can code..."&lt;/h4&gt;&lt;p&gt;&lt;em&gt;That's awesome!&lt;/em&gt; We have projects in a diverse array of languages, using myriad
tools and frameworks. Depending on the skills you have, or are planning to
acquire, you can pick a project and start contributing to it. Based on your
contributions and your familiarity with the codebase, you can then apply for the
role that matches your desired level of involvement.&lt;/p&gt;
&lt;p&gt;So if you want to be lightly involved with code-reviews and would like to know
about our plans in advance, you can start off as a Project Contributor. This is
a fantastic role to get started with and ensures that you get excellent
mentorship as you start your FOSS journey.&lt;/p&gt;
&lt;p&gt;As your familiarity with the codebase increases, you might want to triage
incoming issues or block certain PRs that you've reviewed. You could escalate
your role to Project Collaborator. Want to me more involved? You can apply to be
a Project Core Committer, or even a Project Maintainer.&lt;/p&gt;
&lt;h4 id="no-i-can-t-code..""&gt;"No, I can't code..."&lt;/h4&gt;&lt;p&gt;&lt;em&gt;That's cool too!&lt;/em&gt; We realise that open source communities are never just about
the code. If you're passionate about growing the CC community by enabling new
contributors to get started or by spreading the word, you can apply for one of
the Community Building roles. Like the Project roles, there are a couple of
levels to choose from.&lt;/p&gt;
&lt;p&gt;Community builders have a whole different set of responsibilities and privileges
specifically catered to the unique task of cultivating a healthy community
around our many open source projects.&lt;/p&gt;
&lt;p&gt;So if you want to be lightly involved with onboarding new contributors to the
repositories and the workflows, you could start off as a Community Contributor.
This is a fantastic role to help new contributors get a headstart in their
journey with FOSS.&lt;/p&gt;
&lt;p&gt;As your familiarity with the community increases, you might want to suggest
tweets for our Twitter account, or pariticipate in long-term community building
tasks from Asana. You could escalate your role to Community Collaborator. Want
to me more involved? You can even apply to be a Community Maintainer.&lt;/p&gt;
&lt;h3 id="what-s-next"&gt;What's next?&lt;/h3&gt;&lt;p&gt;The Community Team is a fairly novel idea for us and we're still tweaking things
along the way. For example, we recently merged of two Project roles, namely
Project Member and Project Collaborator, when we realised they weren't so
different. As we internalise these roles more and more, we'll find more scope
for improvement and we'll continue to refine these roles over time.&lt;/p&gt;
&lt;p&gt;We're excited about the Community Team. If you're interested in joining us on
this ride, it's really easy to &lt;a href="/community/community-team/"&gt;get started&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;&lt;small&gt;*We do not guarantee that you will be accepted if you apply for an
internship!&lt;/small&gt;&lt;/p&gt;
</content></entry><entry><title>Accessibility Improvements: Final Changes and Modal Accessilibity</title><link href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week9-10/" rel="alternate"></link><updated>2020-08-12T00:00:00Z</updated><author><name>AyanChoudhary</name></author><id>urn:uuid:4b846809-9764-37d3-9551-f5c639064470</id><content type="html">&lt;p&gt;These are the last two weeks of my internship with CC. I am working on improving the accessibility of cc-search and internationalizing it as well.
This post contains details of my work done to make accessibility improvements to the search result page and the image detail page and also covers some advanced accessiblity improvement details.&lt;/p&gt;
&lt;p&gt;The topics included in this post cover:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Tooltip accessibility and keyboard interactions&lt;/li&gt;
&lt;li&gt;Improve modal accessibility and implement trap focus&lt;/li&gt;
&lt;li&gt;Fix &lt;code&gt;&amp;lt;label&amp;gt;&lt;/code&gt; for form elements&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The first stage involved fixing the license explanation tooltips. These tooltips worked fine on click but did not respond to keypress events.
The solution to overcome this was to use an event listener on the element which would would execute the &lt;code&gt;showLicenseExplanation&lt;/code&gt; function onClick.
Luckily &lt;code&gt;VueJS&lt;/code&gt; provides this function inbuilt via the &lt;code&gt;v-on:keyup&lt;/code&gt; attribute. So after change the code looks as follows:&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;img&lt;/span&gt;
    &lt;span class="na"&gt;:aria-label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;$t(&amp;#39;browse-page.aria.license-explanation&amp;#39;)&amp;quot;&lt;/span&gt;
    &lt;span class="na"&gt;tabindex&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;
    &lt;span class="na"&gt;v-if&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;filterType == &amp;#39;licenses&amp;#39;&amp;quot;&lt;/span&gt;
    &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;@/assets/help_icon.svg&amp;quot;&lt;/span&gt;
    &lt;span class="na"&gt;alt&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;help&amp;quot;&lt;/span&gt;
    &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;license-help is-pulled-right padding-top-smallest padding-right-smaller&amp;quot;&lt;/span&gt;
    &lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="na"&gt;click&lt;/span&gt;&lt;span class="err"&gt;.&lt;/span&gt;&lt;span class="na"&gt;stop&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;toggleLicenseExplanationVisibility(item.code)&amp;quot;&lt;/span&gt;
    &lt;span class="na"&gt;v-on:keyup&lt;/span&gt;&lt;span class="err"&gt;.&lt;/span&gt;&lt;span class="na"&gt;enter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;toggleLicenseExplanationVisibility(item.code)&amp;quot;&lt;/span&gt;
&lt;span class="p"&gt;/&amp;gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Similar change was made to all the tooltips. The reason behind this error was that non-semantic element representation
(i.e. using &lt;code&gt;&amp;lt;div&amp;gt;&lt;/code&gt;, &lt;code&gt;&amp;lt;span&amp;gt;&lt;/code&gt; or &lt;code&gt;&amp;lt;img&amp;gt;&lt;/code&gt; instead of a &lt;code&gt;&amp;lt;button&amp;gt;&lt;/code&gt;) does not register a keypress listener for these tags and hence they don't respond on keypress.&lt;/p&gt;
&lt;p&gt;The second change is related to modals. Modals have some stringent accessilibity parameters that have to be carefully handled.
The criteria are:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;On opening the modal the remaining elements should get disabled.&lt;/li&gt;
&lt;li&gt;The modal should have trap-focus(the user should not exit the modal when using tab to navigate).&lt;/li&gt;
&lt;li&gt;The modal should close on pressing &lt;strong&gt;esc&lt;/strong&gt; or on clicking the overlay.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;To meet the criteria we developed a new &lt;a href="https://github.com/creativecommons/cccatalog-frontend/blob/develop/src/components/AppModal.vue"&gt;modal component&lt;/a&gt;.
This modal has an overlay and closes when we press the &lt;strong&gt;esc&lt;/strong&gt; key or click on the overlay. The modal also disables other elements when it is opened.&lt;/p&gt;
&lt;p&gt;The final task achieved in the modal was the implementation of trap focus. For this we used the &lt;a href="https://github.com/posva/focus-trap-vue"&gt;vue-trap-focus library&lt;/a&gt;
The library exposes a &lt;code&gt;&amp;lt;focus-trap&amp;gt;&lt;/code&gt; component which acts as wrapper to enable focus-trap. The implementation we used was:&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;focus-trap&lt;/span&gt; &lt;span class="na"&gt;:active&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;true&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;modal relative&amp;quot;&lt;/span&gt; &lt;span class="na"&gt;aria-modal&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;true&amp;quot;&lt;/span&gt; &lt;span class="na"&gt;role&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;dialog&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;header&lt;/span&gt;
        &lt;span class="na"&gt;v-if&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;title&amp;quot;&lt;/span&gt;
        &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;modal-header padding-top-bigger padding-left-bigger padding-right-normal padding-bottom-small&amp;quot;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;slot&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;header&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
            &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;h3&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;{{ title }}&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;h3&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;slot&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;button&lt;/span&gt;
            &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;button&amp;quot;&lt;/span&gt;
            &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;close-button has-color-gray is-size-6 is-size-4-touch&amp;quot;&lt;/span&gt;
            &lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="na"&gt;click&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;$emit(&amp;#39;close&amp;#39;)&amp;quot;&lt;/span&gt;
            &lt;span class="na"&gt;:aria-label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;$t(&amp;#39;browse-page.aria.close&amp;#39;)&amp;quot;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;i&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;icon cross&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;/&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;button&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;header&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;slot&lt;/span&gt; &lt;span class="na"&gt;default&lt;/span&gt; &lt;span class="p"&gt;/&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;div&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;focus-trap&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Apart from these the modal also has the &lt;code&gt;aria-modal&lt;/code&gt; attribute and the &lt;code&gt;role="dialog"&lt;/code&gt; attribute.
These attributes direct our screen readers to recognise this component as a modal and declare it whenever the modal opens.&lt;/p&gt;
&lt;p&gt;The last improvement involves using appropriate label tags for the form elements. A lot of elements did not have proper labels or were nested in wrong way.
These elements were fixed and after the fixing the nestings the elements had proper labels which the screen readers were able to identify.
An example a proper input elements with correct label nesting is:&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;label&lt;/span&gt; &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;checkbox&amp;quot;&lt;/span&gt; &lt;span class="na"&gt;:for&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;item.code&amp;quot;&lt;/span&gt; &lt;span class="na"&gt;:disabled&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;block(item)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;input&lt;/span&gt;
        &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;checkbox&amp;quot;&lt;/span&gt;
        &lt;span class="na"&gt;class&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;filter-checkbox margin-right-small&amp;quot;&lt;/span&gt;
        &lt;span class="na"&gt;:id&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;item.code&amp;quot;&lt;/span&gt;
        &lt;span class="na"&gt;:key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;index&amp;quot;&lt;/span&gt;
        &lt;span class="na"&gt;:checked&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;item.checked&amp;quot;&lt;/span&gt;
        &lt;span class="na"&gt;:disabled&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;block(item)&amp;quot;&lt;/span&gt;
        &lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="na"&gt;change&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;onValueChange&amp;quot;&lt;/span&gt;
    &lt;span class="p"&gt;/&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;license-icons&lt;/span&gt; &lt;span class="na"&gt;v-if&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;filterType == &amp;#39;licenses&amp;#39;&amp;quot;&lt;/span&gt; &lt;span class="na"&gt;:license&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;item.code&amp;quot;&lt;/span&gt; &lt;span class="p"&gt;/&amp;gt;&lt;/span&gt;
    {{ $t(item.name) }}
&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;label&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Notice how the input is a child of the &lt;code&gt;&amp;lt;label&amp;gt;&lt;/code&gt; tag which has the &lt;code&gt;for&lt;/code&gt; attribute to point which element it labels.&lt;/p&gt;
&lt;p&gt;Apart from these changes, the eslint configuration of the project were also changed to include a11y-linting for the elments.
We used the &lt;a href="https://github.com/maranran/eslint-plugin-vue-a11y"&gt;eslint-plugin-vue-a11y&lt;/a&gt; to enforce accessibility guidelines for our components via lint checks. 
Furthermore all the aria-labels were internationalized to enforce the i18n standard in our repo that we had setup earlier this summer.&lt;/p&gt;
&lt;p&gt;After all these changes we had the following inprovements in the accessibility scores(computed from lighthouse):&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Browse Page: 76 -&amp;gt; 98 | +22&lt;/li&gt;
&lt;li&gt;Collections Browse Page: 86 -&amp;gt; 96 | +10&lt;/li&gt;
&lt;li&gt;Photo Detail Page: 75 -&amp;gt; 95 | +20&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;And we are officially done with our work for the summer internship. The next blog will be the culmination of this series.&lt;/p&gt;
&lt;p&gt;You can track the work done for these weeks through these PRs:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1072"&gt;Accessibility Improvements&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1121"&gt;setup vue-a11y for eslint&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1123"&gt;Aria labels and internationalization&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1120"&gt;internationalize aria-labels for about page and feedback page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1153"&gt;add trap focus to modals&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The progress of the project can be tracked on &lt;a href="https://github.com/creativecommons/cccatalog-frontend"&gt;cc-search&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;CC Search Accessiblity is my GSoC 2020 project under the guidance of &lt;a href="https://creativecommons.org/author/zackcreativecommons-org/"&gt;Zack Krida&lt;/a&gt; and &lt;a href="https://opensource.creativecommons.org/blog/authors/akmadian/"&gt;Ari Madian&lt;/a&gt;, who is the primary mentor for this project, &lt;a href="https://creativecommons.org/author/annacreativecommons-org/"&gt;Anna Tumadóttir&lt;/a&gt; for helping all along and engineering director &lt;a href="https://creativecommons.org/author/kriticreativecommons-org/"&gt;Kriti
Godey&lt;/a&gt;, have been very supportive.&lt;/p&gt;
</content></entry><entry><title>CC Legal Database: Developing features</title><link href="http://opensource.creativecommons.org/blog/entries/legal-database-features/" rel="alternate"></link><updated>2020-08-07T00:00:00Z</updated><author><name>krysal</name></author><id>urn:uuid:22d82fa8-ac33-3574-89ba-98937b7365f7</id><content type="html">&lt;p&gt;In this post, I want to update the progress on Reimplementing the CC Legal Database site, my Outreachy project. There are several features added over the last month to date.&lt;/p&gt;
&lt;h3 id="submission-forms"&gt;Submission forms&lt;/h3&gt;&lt;p&gt;The first thing I wanted to implement was the respective forms so that anyone can submit a case or article to the database. These forms were slightly modified in the redesign (discussed in the previous articles), so now it has fewer mandatory fields to lower the bar and facilitate the contribution of users.&lt;/p&gt;
&lt;figure style="text-align: center;"&gt;
    &lt;img src="scholarship-form.png" alt="Form to submit an article related to CC licenses" style="border: 1px solid black; width: 60%;"&gt;
    &lt;figcaption&gt;Scholarship form to submit an article.&lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;For the Scholarship form, for example, it is only needed to share your name, email and a link to propose an article related to any of the CC licenses, although the more information you can provide us the better, in any case, each contribution is reviewed by the staff before publishing.&lt;/p&gt;
&lt;h3 id="search"&gt;Search&lt;/h3&gt;&lt;p&gt;The second important task was to allow searching in each of the listings. A basic function to start making use of the exposed information. In the &lt;a href="https://labs.creativecommons.org/caselaw/"&gt;current site&lt;/a&gt;, this function is delegated to an external service, a certain famous search engine. Filtering is now performed in the backend based on the keywords entered by the user, thus returning the reduced list. Later this will be combined with filtering by tags or topics that are associated with each entry (case or scholarship).&lt;/p&gt;
&lt;h3 id="automated-tests"&gt;Automated tests&lt;/h3&gt;&lt;p&gt;While developing the mentioned functionalities I was also in charge of adding automatic unit tests, to ensure that future changes to the code base do not damage already functional parts of the site. This, in addition to giving more confidence to future contributors, they provide value immediately, at the time of writing the tests you should think about possible edge cases, so they allowed me to notice a missing validation in a couple of routes and then correct it.&lt;/p&gt;
&lt;figure style="text-align: center;"&gt;
    &lt;img src="404-page.png" alt="404 page" style="border:1px solid black; width:70%;"&gt;
    &lt;figcaption&gt;Example of page obtained when requesting a case detail that is not published or doesn't exist.&lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;In this process of adding automated tests I wanted them to run on every pull request created, so I learned how to write a GitHub Action with a PostgreSQL service, the DBMS used in this case. Previously, I had already created a job for linting, so I needed to add another one to run in parallel to save time. This service provided by GitHub is pretty cool and useful, it opens up a world of possibilities, from running third party services like &lt;a href="https://github.com/GoogleChrome/lighthouse-ci"&gt;Lighthouse test&lt;/a&gt; to even &lt;a href="https://github.com/gr2m/twitter-together"&gt;send tweets&lt;/a&gt;! If you want to see the GitHub Action file configurated for this project, check it out: &lt;a href="https://github.com/creativecommons/legaldb/blob/31c3002a7860d78f3fdb464150c5c1b2f8bb86fc/.github/workflows/main.yml"&gt;&lt;code&gt;.github/workflows/main.yml&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id="accessibility"&gt;Accessibility&lt;/h3&gt;&lt;p&gt;To check if the site had shortcomings I did the Lighthouse test on the homepage, discovering that there were indeed some issues to tackle. In principle the results were these:&lt;/p&gt;
&lt;figure style="text-align: center;"&gt;
    &lt;img src="lighthouse-before.png" alt="" style="border:1px solid black; width:70%;"&gt;
    &lt;figcaption&gt;Initial Lighthouse test measurements.&lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;The good thing about this test is that it throws up suggestions on how to fix the bugs found, so after adding certain missing attributes and labels, the following results were achieved.&lt;/p&gt;
&lt;figure style="text-align: center;"&gt;
    &lt;img src="lighthouse-after.png" alt="" style="border:1px solid black; width:70%;"&gt;
    &lt;figcaption&gt;Lighthouse test measurements after corrections.&lt;/figcaption&gt;
&lt;/figure&gt;&lt;p&gt;There is still room for improvement but at least we are within a quite acceptable green range.&lt;/p&gt;
&lt;h3 id="other-features-and-tweaks"&gt;Other features and tweaks&lt;/h3&gt;&lt;p&gt;Some other features were implemented but only relevant to our registered users, that is, the Legal Staff. They consist of Django admin customization, such as filtering records by status, and a particular thing requested, the answers of frequently asked questions need to be displayed formatted, so they are now saved as Markdown text and transformed to HTML with style on the public site, showing lists, bold text, links, etc. The admin can also see a preview while editing.&lt;/p&gt;
&lt;h3 id="conclusion"&gt;Conclusion&lt;/h3&gt;&lt;p&gt;After reviewing all done this last month I see significant progress has been made, I have learned many things along the way: more of what Django and its ecosystem offers, about accessibility, continuous integration in Heroku and GitHub, and more. One of the things that makes me most happy is being able to be contributing and being part of an Open Source organization, knowing how it moves and works inside, something I never imagine before.&lt;/p&gt;
&lt;p&gt;Time flies and there are less than two weeks left to finish, so if you want to follow the project here is the repository to suggest improvements or report bugs, or if you prefer something less technical you can join us on the &lt;a href="https://creativecommons.slack.com/channels/cc-dev-legal-database"&gt;slack channel&lt;/a&gt;.&lt;/p&gt;
</content></entry><entry><title>Smithsonian Unit Code Update</title><link href="http://opensource.creativecommons.org/blog/entries/smithsonian-unit-code-update/" rel="alternate"></link><updated>2020-08-03T00:00:00Z</updated><author><name>charini</name></author><id>urn:uuid:23985c56-007a-3707-9f9c-66bde8adae7e</id><content type="html">&lt;h2 id="introduction"&gt;Introduction&lt;/h2&gt;&lt;p&gt;The Creative Commons (CC) Catalog project collects and stores CC licensed images scattered across the internet, such
that they can be made accessible to the general public via the &lt;a href="https://ccsearch.creativecommons.org/"&gt;CC Search&lt;/a&gt; and &lt;a href="https://api.creativecommons.engineering/v1/"&gt;CC Catalog API&lt;/a&gt;
tools. Numerous information associated with each image, which help in the image search and categorisation process are
stored via CC Catalog in the CC database.&lt;/p&gt;
&lt;p&gt;In my &lt;a href="/blog/entries/flickr-sub-provider-retrieval/"&gt;previous blog post&lt;/a&gt; of this series entitled 'Flickr Sub-provider Retrieval', I discussed how
the images from a certain provider (such as Flickr) can be categorised based on the sub-provider values (which reflects
the underlying organisation or entity that published the images through the provider). We have similarly implemented
the sub-provider retrieval logic for Europeana and Smithsonian providers. Unlike in Flickr and Europeana, every single
image from Smithsonian is categorised under some sub-provider value where the sub-providers are identified based on a
unit code value as contained in the API response (for more information please refer to the pull request &lt;a href="https://github.com/creativecommons/cccatalog/pull/455"&gt;#455&lt;/a&gt;).
The unit code values and the corresponding sub provider values are maintained in the dictionary
&lt;em&gt;SMITHSONIAN_SUB_PROVIDERS&lt;/em&gt;. However, there is the possibility of the &lt;em&gt;unit code&lt;/em&gt; values being updated at the
Smithsonian API level, and it is important that we have a mechanism of reflecting those updates in the
&lt;em&gt;SMITHSONIAN_SUB_PROVIDERS&lt;/em&gt; dictionary as well. In this blog post, we discuss how we learn the potential
changes to the &lt;em&gt;unit code&lt;/em&gt; values and keep the &lt;em&gt;SMITHSONIAN_SUB_PROVIDERS&lt;/em&gt; dictionary up-to-date.&lt;/p&gt;
&lt;h2 id="implementation"&gt;Implementation&lt;/h2&gt;&lt;h3 id="retrieving-the-latest-unit-codes"&gt;Retrieving the latest unit codes&lt;/h3&gt;&lt;p&gt;We are required to obtain the latest &lt;em&gt;unit codes&lt;/em&gt; supported by the Smithsonian API to achieve this task. Furthermore,
since we are only interested in image data, the &lt;em&gt;unit codes&lt;/em&gt; which are associated with images alone need to be
retrieved. The latest Smithsonian &lt;em&gt;unit codes&lt;/em&gt; corresponding to images can be retrieved by calling the end point
&lt;a href="https://api.si.edu/openaccess/api/v1.0/terms/unit_code?q=online_media_type:Images&amp;amp;api_key=REDACTED"&gt;https://api.si.edu/openaccess/api/v1.0/terms/unit_code?q=online_media_type:Images&amp;amp;api_key=REDACTED&lt;/a&gt;&lt;/p&gt;
&lt;h3 id="check-for-unit-code-updates"&gt;Check for unit code updates&lt;/h3&gt;&lt;p&gt;In order to identify whether changes have occurred to the collection of &lt;em&gt;unit codes&lt;/em&gt; supported by the Smithsonian API
(in the form of additions and/or deletions), we compare the values retrieved by calling the previously mentioned
endpoint, with the values contained in the &lt;em&gt;SMITHSONIAN_SUB_PROVIDERS&lt;/em&gt; dictionary. All changes are reflected in a table
named &lt;em&gt;smithsonian_new_unit_codes&lt;/em&gt; which contains the two fields, 'new_unit_code' and 'action'. If a new &lt;em&gt;unit code&lt;/em&gt; is
introduced at the API level, we store that &lt;em&gt;unit code&lt;/em&gt; value with the corresponding action value 'add' in the table.
This reflects that the given &lt;em&gt;unit code&lt;/em&gt; value needs to be added to the &lt;em&gt;SMITHSONIAN_SUB_PROVIDERS dictionary&lt;/em&gt;. If a
&lt;em&gt;unit code&lt;/em&gt; that appears in the &lt;em&gt;SMITHSONIAN_SUB_PROVIDERS&lt;/em&gt; dictionary does not appear at the API level, we store
the &lt;em&gt;unit code&lt;/em&gt; value with the corresponding action value 'delete' in the table, reflecting that it needs to be deleted
from the dictionary.&lt;/p&gt;
&lt;h3 id="triggering-the-unit-code-update-workflow"&gt;Triggering the unit code update workflow&lt;/h3&gt;&lt;p&gt;A separate workflow named &lt;em&gt;check_new_smithsonian_unit_codes_workflow&lt;/em&gt; allows executing the logic we discussed via the
Airflow UI. For each execution, the table &lt;em&gt;smithsonian_new_unit_codes&lt;/em&gt; is completely cleared of previous data, and the
latest updates to reflect in the &lt;em&gt;SMITHSONIAN_SUB_PROVIDERS&lt;/em&gt; dictionary are stored. Note that the actual updates to
the dictionary (as reflected in the table) needs to be carried out by a person, since editing the dictionary is not
automated. Furthermore, this workflow is expected to be executed at-least once a week, preferably prior to running
the Smithsonian image retrieval script such that the Smithsonian sub-provider retrieval task can be run with no issue.&lt;/p&gt;
&lt;h2 id="acknowledgement"&gt;Acknowledgement&lt;/h2&gt;&lt;p&gt;I express my gratitude to my GSoC supervisor Brent Moran for assisting me with this task.&lt;/p&gt;
</content></entry><entry><title>Linked Commons: Autocomplete Feature</title><link href="http://opensource.creativecommons.org/blog/entries/linked-commons-autocomplete-feature/" rel="alternate"></link><updated>2020-07-31T00:00:00Z</updated><author><name>subhamX</name></author><id>urn:uuid:0e278e85-748f-3d35-a640-24daab837875</id><content type="html">&lt;p&gt;The following blog intends to explain the very recent feature integrated to the Linked Commons. Be it the giant Google Search or any small website having a form field, everyone wishes to predict what’s on the user’s mind. For every keystroke, a nice search bar always renders some possible options the user could be looking for. The core ideology behind having this feature is — &lt;em&gt;do as much work as possible for the user!&lt;/em&gt;&lt;/p&gt;
&lt;div style="text-align: center; width: 100%;"&gt;
    &lt;figure&gt;
        &lt;img src="autocomplete-feat-in-action.gif" alt="autocomplete-feature" style="border: 1px solid black"&gt;
        &lt;figcaption style="font-weight: 500;"&gt;Autocomplete feature in action&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;h2 id="motivation"&gt;Motivation&lt;/h2&gt;&lt;p&gt;One of the newest features integrated last month into Linked Commons is Filtering by node name. Here a user can search for his/her favourite node and explore all its neighbours. Since the list is very big, it was self-evident for us to have a text box (and not a drop-down) where the user is supposed to type the node name.&lt;/p&gt;
&lt;p&gt;Some of the reasons why to have "autocomplete feature" in the filtering by node name -&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Some of the node names are very uncommon and lengthy. There is a high probability of misspelling it.&lt;/li&gt;
&lt;li&gt;Submitting the form and getting a response of “Node doesn’t exist” isn’t a very good user flow, and we want to minimise such incidents.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Also, on a side note giving a search bar to the user and giving no hints is ruthless. We all need recommendations and guess what linked commons got you covered! Now for every keystroke, we load a bunch of node names which you might be looking for. ;)&lt;/p&gt;
&lt;h2 id="problem"&gt;Problem&lt;/h2&gt;&lt;p&gt;The autocomplete feature on a very basic level aims to predict the rest of a word the user is typing. A possible implementation is though the linear traversal of all the nodes in the list. It will be having a &lt;strong&gt;linear time complexity&lt;/strong&gt;. It’s not very good and it’s very obvious to look for a faster and more efficient way. Also, even if for once we neglect the &lt;strong&gt;time complexity&lt;/strong&gt;, looking for the best 10 nodes out of these millions on the client's machine is not at all a good idea; it will cause throttling and will result in performance drops. 
On the other hand, a &lt;strong&gt;trie based solution&lt;/strong&gt; is more efficient for sure but still, we cannot do this indexing on the client machine for the same reasons stated above.
So far, it is now apparent that we implement this feature on the server and also aim for at least something better than linear time complexity.&lt;/p&gt;
&lt;h2 id="a-non-conventional-solution"&gt;A non-conventional solution&lt;/h2&gt;&lt;p&gt;We could have used Elastic Search, which is very powerful and has a ton of functionalities but since our needs are very small we wanted to look for other simple alternatives. Moreover, we didn't want to complicate our current architecture by adding an additional framework and libraries.&lt;/p&gt;
&lt;p&gt;Taking the above points into consideration we went ahead with the following solution. We store all nodes data into an SQL dB and search for all the nodes whose domain name pattern was matching to the query string. After slicing the query set and other randomization we sent the payload to the client. To make it more robust, we are caching the results in the frontend to avoid multiple calls for the same query. It will surely reduce the load from the server and also give a faster response.&lt;/p&gt;
&lt;h2 id="results"&gt;Results&lt;/h2&gt;&lt;p&gt;To make sure our solution works well, we performed load tests, checking that any response time does not exceed 1000 ms. We used locust which is a user load testing tool. We simulated with &lt;strong&gt;1000 users&lt;/strong&gt; and &lt;strong&gt;10 as Hatch rate&lt;/strong&gt;. 
The following test is performed on the local machine to ensure that the server location isn’t affecting the results.&lt;/p&gt;
&lt;p&gt;Here are some aggregated result statistics.&lt;/p&gt;
&lt;table class="table table-striped"&gt;
&lt;thead class="thead-dark"&gt;&lt;tr&gt;
&lt;th&gt;Field Name&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Request Count&lt;/td&gt;
&lt;td&gt;&lt;strong&gt; 23323     &lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failure Count&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;  0        &lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Median Response Time&lt;/td&gt;
&lt;td&gt;&lt;strong&gt; 360 ms    &lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Average Response Time&lt;/td&gt;
&lt;td&gt;&lt;strong&gt; 586.289 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Min Response Time&lt;/td&gt;
&lt;td&gt;&lt;strong&gt; 4.03094 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max Response Time&lt;/td&gt;
&lt;td&gt;&lt;strong&gt; 4216 ms   &lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Average Content Size&lt;/td&gt;
&lt;td&gt;&lt;strong&gt; 528.667 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Requests/s&lt;/td&gt;
&lt;td&gt;&lt;strong&gt; 171.754   &lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max Requests/s&lt;/td&gt;
&lt;td&gt;&lt;strong&gt; 214       &lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failures/s&lt;/td&gt;
&lt;td&gt;&lt;strong&gt; 0         &lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="next-steps"&gt;Next steps&lt;/h2&gt;&lt;p&gt;In the next blog, we will be covering the long awaited data update and the new architecture.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;&lt;p&gt;Overall, I enjoyed working on this feature and it was a great learning experience. This feature has been successfully integrated to the development version, do check it out. Now that you have read this blog till the end, I hope that you enjoyed it. For more information please visit our &lt;a href="https://github.com/creativecommons/cccatalog-dataviz/"&gt;Github repo&lt;/a&gt;. We are looking forward to hearing from you on linked commons. Our &lt;a href="https://creativecommons.slack.com/channels/cc-dev-cc-catalog-viz"&gt;slack&lt;/a&gt; doors are always open to you, see you there. :)&lt;/p&gt;
</content></entry><entry><title>CC Search, Initial Accessibility Improvements</title><link href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week7-8/" rel="alternate"></link><updated>2020-07-25T00:00:00Z</updated><author><name>AyanChoudhary</name></author><id>urn:uuid:b523aa13-3429-3303-b7cb-3c66beeed723</id><content type="html">&lt;p&gt;These are the seventh and eighth weeks of my internship with CC. I am working on improving the accessibility of cc-search and internationalizing it as well.
This post contains details of my work done to make initial accessibility improvements to homepage and the other static pages.&lt;/p&gt;
&lt;p&gt;With the internationalization work complete, our next target were the accessiblity improvements. So I decided to tackle the homepage and the static pages first.
The aforementioned pages had the following accessiblity issues:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;No aria-label on links&lt;/li&gt;
&lt;li&gt;Improper landmarks&lt;/li&gt;
&lt;li&gt;Improper aria-control nestings&lt;/li&gt;
&lt;li&gt;Some elements not being read by the screen-reader&lt;/li&gt;
&lt;li&gt;Color contrast Issues(to be covered later)&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;But before working I ran another set of audit tests to exactly pin-point these issues. I used &lt;a href="https://www.nvaccess.org/"&gt;NVDA&lt;/a&gt; for running these audits.
Lets go through the fixes one at a time.&lt;/p&gt;
&lt;p&gt;The first issue of no aria-label was pre-dominantly found in the footer. we had some links such as:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;&amp;lt;a
    href="https://www.instagram.com/creativecommons"
    class="social has-text-white"
    target="_blank"
    rel="noopener"
&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;These links did not contain any aria-label and were read as &lt;strong&gt;cc link&lt;/strong&gt;. So the aria-labels had to be added &lt;code&gt;aria-label="instagram link"&lt;/code&gt; in this case which fixed this problem.&lt;/p&gt;
&lt;p&gt;The next issue was of improper landmarks. Most of the pages had no &lt;strong&gt;main&lt;/strong&gt; landmark and some had no &lt;strong&gt;complimentary&lt;/strong&gt; or &lt;strong&gt;region&lt;/strong&gt; landmarks even though they were required in those pages.
These landmarks had to be added after the carefully scrutinising the pages in the audits.&lt;/p&gt;
&lt;p&gt;The next issue was of improper aria-control nestings. This is interesting as it involves having some deeper understanding of the roles involved. So I will explain this in a little depth.
The area where we had this issue was in feedback page. The code involved was:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;&amp;lt;ul&amp;gt;
    &amp;lt;li :class="tabClass(0, 'tab')"&amp;gt;
        &amp;lt;a
            href="#panel0"
            :aria-selected="activeTab == 0"
            @click.prevent="setActiveTab(0)"
        &amp;gt;
        Help us Improve
        &amp;lt;/a&amp;gt;
    &amp;lt;/li&amp;gt;
    &amp;lt;li :class="tabClass(1, 'tab')"&amp;gt;
        &amp;lt;a
            href="#panel1"
            :aria-selected="activeTab == 1"
            @click.prevent="setActiveTab(1)"
        &amp;gt;
        Report a Bug
        &amp;lt;/a&amp;gt;
    &amp;lt;/li&amp;gt;
&amp;lt;/ul&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The reason why this is an error is because of the &lt;code&gt;aria-selected&lt;/code&gt; attribute can only be applied to an element having the role &lt;strong&gt;tab&lt;/strong&gt; nested inside a &lt;strong&gt;tablist&lt;/strong&gt; element.
For reference, in the above example the &lt;code&gt;&amp;lt;ul&amp;gt;&lt;/code&gt; should have the role &lt;strong&gt;tablist&lt;/strong&gt; and each &lt;code&gt;&amp;lt;li&amp;gt;&lt;/code&gt; element should have the role &lt;strong&gt;tab&lt;/strong&gt;. And so the &lt;code&gt;aria-selected&lt;/code&gt; attribute should be in the &lt;code&gt;&amp;lt;li&amp;gt;&lt;/code&gt; element instead of the &lt;code&gt;&amp;lt;a&amp;gt;&lt;/code&gt; tag.&lt;/p&gt;
&lt;p&gt;The corrected code is:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;&amp;lt;ul role="tablist"&amp;gt;
    &amp;lt;li role="tab" :class="tabClass(0, 'tab')" :aria-selected="activeTab == 0"&amp;gt;
        &amp;lt;a 
            aria-label="help us improve form" 
            href="#panel0"
            @click.prevent="setActiveTab(0)"
        &amp;gt;
        {{ $t('feedback.improve') }}
        &amp;lt;/a&amp;gt;
    &amp;lt;/li&amp;gt;
    &amp;lt;li role="tab" :class="tabClass(1, 'tab')" :aria-selected="activeTab == 1"&amp;gt;
        &amp;lt;a 
            aria-label="report a bug form"
            href="#panel1"
            @click.prevent="setActiveTab(1)"
        &amp;gt;
        {{ $t('feedback.bug') }}
        &amp;lt;/a&amp;gt;
    &amp;lt;/li&amp;gt;
&amp;lt;/ul&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Another interesting finding involved the screen readers not reading particular special characters such as &lt;code&gt;~&lt;/code&gt; and &lt;code&gt;|&lt;/code&gt;.
This issue was quite pronounced in the search guide page where these symbols were used in plenty in both links as well as texts.
So I had to phonetically write these out in the aria-labels of the links to make the screen reader read them out loud.
The corresponding changes are:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;&amp;lt;a
    aria-label="dog vertical bar cat"
    href="https://search.creativecommons.org/search?q=dog%7Ccat"
&amp;gt;
    &amp;lt;em&amp;gt;dog|cat&amp;lt;/em&amp;gt;
&amp;lt;/a&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;After all these changes we had some increase in the accessibility scores(computed from lighthouse):&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;About Page: 78 -&amp;gt; 97 | +19&lt;/li&gt;
&lt;li&gt;Search-Guide Page: 76 -&amp;gt; 97 | +23&lt;/li&gt;
&lt;li&gt;Feedback Page: 75 -&amp;gt; 97 | +22&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Whoosh!! That was quite a lot. We are done with these two weeks for now. Hope to see you in the next post as well.&lt;/p&gt;
&lt;p&gt;You can track the work done for these weeks through these PRs:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1068"&gt;Accessibility&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1072"&gt;Accessibility Improvements&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The progress of the project can be tracked on &lt;a href="https://github.com/creativecommons/cccatalog-frontend"&gt;cc-search&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;CC Search Accessiblity is my GSoC 2020 project under the guidance of &lt;a href="https://creativecommons.org/author/zackcreativecommons-org/"&gt;Zack Krida&lt;/a&gt; and &lt;a href="https://opensource.creativecommons.org/blog/authors/akmadian/"&gt;Ari Madian&lt;/a&gt;, who is the primary mentor for this project, &lt;a href="https://creativecommons.org/author/annacreativecommons-org/"&gt;Anna Tumadóttir&lt;/a&gt; for helping all along and engineering director &lt;a href="https://creativecommons.org/author/kriticreativecommons-org/"&gt;Kriti
Godey&lt;/a&gt;, have been very supportive.&lt;/p&gt;
</content></entry><entry><title>Data flow: from API to DB</title><link href="http://opensource.creativecommons.org/blog/entries/data-flow-API-to-DB/" rel="alternate"></link><updated>2020-07-22T00:00:00Z</updated><author><name>srinidhi</name></author><id>urn:uuid:44db39c0-d562-3a62-b5c5-ac4babcbbe40</id><content type="html">&lt;h2 id="introduction"&gt;Introduction&lt;/h2&gt;&lt;p&gt;The CC Catalog project  handles the flow of image metadata from the source or
provider and loads it to the database, which is then surfaced to the &lt;a href="https://ccsearch.creativecommons.org/about"&gt;CC
search&lt;/a&gt; tool. The workflows are set up for each provider to gather
metadata about CC licensed images. These workflows are handled with the help of
Apache Airflow. Airflow is an open source tool that helps us to schedule and
monitor workflows.&lt;/p&gt;
&lt;h2 id="airflow-intro"&gt;Airflow intro&lt;/h2&gt;&lt;p&gt;Apache Airflow is an open source tool that helps us to schedule tasks and
monitor workflows . It provides an easy to use UI that makes managing tasks
easy.  In Airflow, the tasks we want to schedule are organised in DAGs
(Directed Acyclic Graphs). DAGs consist of a collection of tasks, and a
relationship defined among these tasks, so that they run in an organised
manner. DAGs files are standard python files that are loaded from  the defined
&lt;code&gt;DAG_FOLDER&lt;/code&gt; on a host. Airflow selects all the python files in the
&lt;code&gt;DAG_FOLDER&lt;/code&gt; that have a DAG instance defined globally, and executes them to
create the DAG objects.&lt;/p&gt;
&lt;h2 id="cc-catalog-workflow"&gt;CC Catalog Workflow&lt;/h2&gt;&lt;p&gt;In the CC catalog, Airflow is set up inside a docker container along with other
services . The loader and provider workflows are inside the &lt;code&gt;dags&lt;/code&gt; directory in
the repo &lt;a href="https://github.com/creativecommons/cccatalog/tree/dacb48d24c6ae9b532ff108589b9326bde0d37a3/src/cc_catalog_airflow/dags"&gt;dag folder&lt;/a&gt;. Provider workflows are set up to pull metadata
about CC licensed images from the respective providers , the data pulled is
structured into a standardised format and written into a TSV (Tab Separated
Values) file locally. These TSV files are then loaded into S3 and then finally
to PostgreSQL DB by the loader workflow.&lt;/p&gt;
&lt;h2 id="provider-api-workflow"&gt;Provider API workflow&lt;/h2&gt;&lt;p&gt;The provider workflows are usually scheduled in one of two time frequencies,
daily or monthly.&lt;/p&gt;
&lt;p&gt;Providers such as Flickr or Wikimedia Commons that are filtered using the date
parameter are usually scheduled for daily jobs. These providers have a large
volume of continuously changing data, and so daily updates are required to keep
the data in sync.&lt;/p&gt;
&lt;p&gt;Providers that are scheduled for monthly ingestion are ones with a relativley
low volume of data, or for which filtering by date is not possible. This means
we need to ingest the entire collection at once. Examples are museum providers
like the &lt;a href="https://collection.sciencemuseumgroup.org.uk/"&gt;Science museum UK&lt;/a&gt; or &lt;a href="https://www.smk.dk/"&gt;Statens Museum for
Kunst&lt;/a&gt;. We don’t expect museum providers to change data on a daily basis.&lt;/p&gt;
&lt;p&gt;The scheduling of the DAGs by the scheduler daemons depends on a few
parameters.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;start_date&lt;/code&gt; - it denotes the starting date from which the
task should begin running. &lt;/li&gt;
&lt;li&gt;&lt;code&gt;schedule_interval&lt;/code&gt; - it denotes the interval between subsequent runs, it
can be specified with airflow keyword strings like “@daily”, “@weekly”,
“@monthly”, “@yearly” other than these we can also schedule the interval using
cron expression.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Example: Cleveland museum is currently scheduled for a monthly crawl with a
starting date as &lt;code&gt;2020-01-15&lt;/code&gt;. &lt;a href="https://github.com/creativecommons/cccatalog/blob/dacb48d24c6ae9b532ff108589b9326bde0d37a3/src/cc_catalog_airflow/dags/cleveland_museum_workflow.py"&gt;cleveland_museum_workflow&lt;/a&gt;&lt;/p&gt;
&lt;h2 id="loader-workflow"&gt;Loader workflow&lt;/h2&gt;&lt;p&gt;The data from the provider scripts are not directly loaded into S3. Instead,
they are stored in a TSV file on the local disk, and the tsv_postgres workflow
handles loading of data to S3, and eventually PostgreSQL. The DAG starts by
calling the task to stage the oldest tsv file from the output directory of the
provider scripts to the staging directory. Next, two tasks run in parallel, one
loads the tsv file in the staging directory to S3 , while the other creates the
loading table in the PostgreSQL database. Once the data is loaded to S3 and the
loading table has been created, the data from S3 is loaded to the intermediate
loading table and then finally inserted into the image table. If loading from
S3 fails the data is loaded to PostgreSQL from the locally stored tsv file.
When the data has been successfully transferred to the image table, the
intermediate loading table is dropped and the tsv files in the staging
directory are deleted. If the copying the tsv files to S3 fails or then those
files are moved to the failure directory for future inspection.&lt;/p&gt;
&lt;div style="text-align:center;"&gt;
    &lt;img src="loader_workflow.png" width="1000px"/&gt;
    &lt;p&gt; Loader workflow &lt;/p&gt;
&lt;/div&gt;&lt;h2 id="acknowledgement"&gt;Acknowledgement&lt;/h2&gt;&lt;p&gt;I would like to thank Brent Moran for helping me write this blog post.&lt;/p&gt;
</content></entry><entry><title>What is up? - CCOS Revamp</title><link href="http://opensource.creativecommons.org/blog/entries/what-is-up-ccos/" rel="alternate"></link><updated>2020-07-20T00:00:00Z</updated><author><name>dhruvi16</name></author><id>urn:uuid:db2b95cd-c221-3f56-9ffb-8b165676fdbe</id><content type="html">&lt;p&gt;In my previous blog, I demonstrated what my Outreachy project was about. Here I will talk about my progress in the past 7 weeks.&lt;/p&gt;
&lt;h3 id="the-set-up"&gt;The Set-Up -&lt;/h3&gt;&lt;p&gt;The &lt;a href="https://opensource.creativecommons.org/"&gt;Creative Commons Open Source&lt;/a&gt; website is built using &lt;a href="https://www.getlektor.com/"&gt;Lektor&lt;/a&gt;. I was not very familiar with it, so I started by going through the documentation and the official website code. I learned how awesome it is and can also be used by non-coders. I got familiar with the &lt;a href="https://palletsprojects.com/p/jinja/"&gt;jinja templates&lt;/a&gt; and working of themes in a Lektor app. For integrating new styles from Vocabulary, I replaced &lt;code&gt;templates/&lt;/code&gt; folder with a &lt;code&gt;theme/&lt;/code&gt; folder. Here is the link to how &lt;a href="https://www.getlektor.com/docs/templates/"&gt;templates&lt;/a&gt; work in Lektor.&lt;/p&gt;
&lt;p&gt;As the revamping process is gradual, there was a need of setting up a staging environment where we could test the website. Deploying the branch that consists of the ongoing changes was pretty easy, I just followed the official &lt;a href="https://www.netlify.com/blog/2016/05/25/lektor-on-netlify-a-step-by-step-guide/"&gt;documentation&lt;/a&gt; provided by Netlify and deployed it.&lt;/p&gt;
&lt;h3 id="adding-new-components-to-vocabulary"&gt;Adding New Components to Vocabulary -&lt;/h3&gt;&lt;p&gt;The &lt;a href="https://www.figma.com/file/mttcugI1UxvCJRKE5sdpbO/Mockups"&gt;mock-ups&lt;/a&gt; for the new CCOS website extensively use Vocabulary components, styles, and patterns, and it had components that were not available in Vocabulary. So, I worked on building them from scratch. I enjoyed this part a bit too much. And also this was a part of the project, I did not think would take up like 2 weeks but it did. I enjoyed questioning the scope, the design, the experience of the components, and getting satisfactory answers. Maintaining the practices, focusing on details were fun things to do. It felt like I own those components. You can check them out &lt;a href="https://cc-vocabulary.netlify.app/?path=/docs/vocabulary-introduction--page"&gt;here&lt;/a&gt; and also use them wherever needed.&lt;/p&gt;
&lt;h3 id="updating-templates-of-the-theme"&gt;Updating Templates of the Theme -&lt;/h3&gt;&lt;p&gt;I started by updating the home page template. I try to make the code cleaner and more readable. Going through the &lt;a href="https://www.getlektor.com/"&gt;Lektor&lt;/a&gt; documentation, I came across different ways to do so. One of them was &lt;a href="/blog/entries/what-is-up-ccos/(https://www.getlektor.com/"&gt;flow blocks&lt;/a&gt;), I like how it makes a template more modular and readable so I implemented the home page using flow blocks. This one after one, I started updating every template. For now, I have updated 10 templates and I plan to update the remaining in upcoming weeks.&lt;/p&gt;
&lt;h3 id="my-experience-so-far"&gt;My Experience so far -&lt;/h3&gt;&lt;p&gt;This has been one heck of a journey for me. I have never collaborated with such a huge open-source organization and so that was something new for me. I have learned a lot of things both technical and non-technical so far. I have become more alert about the code I write, this journey has helped me improve the questions I ask to myself while writing code or thinking about the solution, I got to learn about new technologies such as &lt;a href="https://www.getlektor.com/"&gt;Lektor&lt;/a&gt;, &lt;a href="https://webpack.js.org/"&gt;Webpack&lt;/a&gt;, &lt;a href="https://sass-lang.com/documentation/syntax"&gt;SCSS&lt;/a&gt; and many more. I am just very glad to be a part of this.&lt;/p&gt;
</content></entry><entry><title>Linked Commons: What's new?</title><link href="http://opensource.creativecommons.org/blog/entries/linked-commons-whats-new/" rel="alternate"></link><updated>2020-07-16T00:00:00Z</updated><author><name>subhamX</name></author><id>urn:uuid:cf56c986-4df1-344a-a15e-e852c66f895d</id><content type="html">&lt;p&gt;&lt;strong&gt;Linked Commons&lt;/strong&gt; is a visualization project which aims to showcase and establish a relationship between millions of data points of licensed content metadata using graphs. Since it is the first blog of this new series, let’s discuss the core ideology of having this project and then about new features. Development of all components mentioned in this blog is complete and has been successfully integrated, so do check out the development version. Happy Reading!&lt;/p&gt;
&lt;h2 id="motivation-and-why-does-visualization-matter"&gt;Motivation and why does visualization matter?&lt;/h2&gt;&lt;p&gt;The number of websites using creative commons licensed content is very huge and growing very rapidly. The CC catalog hosts these millions of data points and each node contains information about the URL of the websites and the licenses used. One can surely do rigorous data analysis, but this would only be interpretable by a few people with a technical background.  On the other hand,  by visualizing data, it becomes incredibly easier to identify patterns and trends. As an old saying, a picture is worth a thousand words. That’s the core ideology of the Linked Commons, i.e. to show the millions of licensed content metadata and how these nodes are connected in a visually appealing form on the HTML canvas like a picture.&lt;/p&gt;
&lt;h2 id="task-1-code-refactoring"&gt;Task 1: Code Refactoring&lt;/h2&gt;&lt;p&gt;My first task was to refactor the code and migrate it to react. The existing codebase had all core functionalities, but we wanted to make it more modular, improve the design, code readability, and reduce complexity. This will help us maintain this project in the long run. Also, it will be easier for the community to contribute and understand the logic.&lt;/p&gt;
&lt;h2 id="task-2-graph-filtering"&gt;Task 2: Graph Filtering&lt;/h2&gt;&lt;h3 id="need-for-filtering-methods"&gt;Need for Filtering Methods&lt;/h3&gt;&lt;div style="text-align: center; width: 90%; margin-left: 5%;"&gt;
    &lt;figure&gt;
        &lt;img src="big-graph.png" alt="Large Graph" style="border: 1px solid black"&gt;
        &lt;figcaption style="font-weight: 500;"&gt;Pic showing clustors of a Graph with 9982 nodes 5000 links&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;p&gt;The aggregate data that cc catalog has is in hundreds of millions. Rendering a graph with these many nodes will be a nightmare for the browser’s rendering and JavaScript engine. Just like we divide any standard textbook into chapters, we thought about adding filtering options that enable the user to retrieve precise information according to certain criteria selected by them. Hence,  we need to have a way in which we can filter the aggregate data into smaller chunks.&lt;/p&gt;
&lt;h3 id="what-filtering-methods"&gt;What filtering methods?&lt;/h3&gt;&lt;p&gt;After brainstorming for a while, we converged and agreed to have &lt;strong&gt;filtering based on node name and distance&lt;/strong&gt;. The primary reason behind this was, it is kind of basic that a person would like to look for his/her favourite node and its neighbours. This is not the end for sure, and many more filtering methods will be added, maybe with the support of chaining one after another. This is just a baby step!&lt;/p&gt;
&lt;h3 id="server-side-filtering-vs-client-side-filtering"&gt;Server-side Filtering vs Client-Side Filtering?&lt;/h3&gt;&lt;p&gt;Now that we know on what query params the filter should work, we need to decide where to do the filtering. Should we do it on the client machine or do it on our server and pass the processed and filtered data to the client? In any filtering method, we need to traverse the whole graph. The JS engine in the browser is doing rendering stuff, complex calculation, etc. With all these processes, doing a full traversal of the dataset having more than a million nodes is going to take a lot of time and memory. The above claim assumes that we have a moderately dense graph. On the other hand, another strategy to accomplish graph filtering could be to delegate that load to a server, and the client’s browser can ask for a fresh copy of the filtered data whenever needed. As mentioned above the client-side filtering has serious shortcomings and user experience won’t be very good with browser freezing and frame drops. So, that's why we decided to go with the latter option i.e server-side filtering.&lt;/p&gt;
&lt;div style="text-align: center;"&gt;
    &lt;figure&gt;
        &lt;img src="filtering-in-action.gif" alt="Filtering In Action" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;Filtering In Action&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;h2 id="task-3-new-design"&gt;Task 3: New Design&lt;/h2&gt;&lt;p&gt;My third task was to upgrade the front-end design of the project. It now has a very clean and refreshing look along with the support for both light and dark theme. Check out our webpage in dark mode and do let us know if it saves your PC energy consumption (As claimed by some websites). Now you all can visit the Linked Commons webpage at mid-night too, no strain to the eyes. ;)&lt;/p&gt;
&lt;div style="text-align: center; width: 90%; margin-left: 5%;"&gt;
    &lt;figure&gt;
        &lt;img src="new-design-light.png" alt="Light Theme" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;Linked Commons - Light Theme&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;h2 id="next-steps"&gt;Next steps&lt;/h2&gt;&lt;p&gt;In the next two weeks, I will be working on the following features.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Implement suggestions API on the server and integrate it with the frontend&lt;/li&gt;
&lt;li&gt;Update the visualization with a more recent and bigger dataset&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;&lt;p&gt;Overall, it was fantastic and rejuvenating experience working on these tasks. Now that you have read this blog till the end, I hope that you enjoyed it. For more information visit our &lt;a href="https://github.com/creativecommons/cccatalog-dataviz/"&gt;Github repo&lt;/a&gt;. We are looking forward to hearing from you on linked commons. Our &lt;a href="https://creativecommons.slack.com/channels/cc-dev-cc-catalog-viz"&gt;slack&lt;/a&gt; doors are always open to you, see you there. :)&lt;/p&gt;
</content></entry><entry><title>Internationalization continued: Modifying tests</title><link href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week5-6/" rel="alternate"></link><updated>2020-07-10T00:00:00Z</updated><author><name>AyanChoudhary</name></author><id>urn:uuid:91d205ed-b94e-3d2a-9c30-5af7f04506ac</id><content type="html">&lt;p&gt;These are the fifth and sixth weeks of my internship with CC. I am working on improving the accessibility of cc-search and internationalizing it as well.
This post contains yet another important aspect to be taken care of while internationalizing the Vue components, i.e. modifying tests to include the changes.&lt;/p&gt;
&lt;p&gt;The components which were left are the two pages displaying the most content:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/blob/develop/src/pages/BrowsePage.vue"&gt;Browse page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/blob/develop/src/pages/PhotoDetailPage.vue"&gt;ImageDetail page&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The above two pages too were handled similar to the remaining pages, special care had to be taken in case of ImageDetail page since there are too many components in different files.
By this point we also have our json structure figured out, mostly and are ready to push the json for fetching further translations.&lt;/p&gt;
&lt;p&gt;Now let's look at the modifications required in the tests. We generally use &lt;code&gt;$t&lt;/code&gt; to access strings from the locales json, but this method/custom-component is not present in the testing vue instance, so we had to inject this method usin localVue and a custom i18n instance.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;const localVue = createLocalVue();
localVue.use(Vuex);
localVue.use(VueI18n);
const messages = require('@/locales/en.json');

const i18n = new VueI18n({
  locale: 'en',
  fallbackLocale: 'en',
  messages,
});
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now we inject this 18n component into our vue instance and we have access to our &lt;code&gt;$t&lt;/code&gt;, but there is still one more step left.
We still need to mock its functionality in the tests. So we create a mock &lt;code&gt;$t&lt;/code&gt; instance to mock in our component. The final code is given below:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;const $t = (key) =&amp;gt; i18n.messages[key];

options = {
    mocks: {
        $t,
    },
};
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Now we are ready to render our component using these custom options with mocks for testing.&lt;/p&gt;
&lt;p&gt;And &lt;em&gt;drum rolls&lt;/em&gt; we have successfully completed Internationalization of the complete cc search. Below are the images for some of the completed pages:&lt;/p&gt;
&lt;p&gt;&lt;img src="/blog/entries/cc-search-accessibility-week5-6/final.png" alt="final.png"&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src="/blog/entries/cc-search-accessibility-week5-6/finalAbout.png" alt="finalAbout.png"&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src="/blog/entries/cc-search-accessibility-week5-6/finalImageDetail.png" alt="finalImageDetail.png"&gt;&lt;/p&gt;
&lt;p&gt;The issues closed with the completion of Internationalization are:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/issues/487"&gt;[META] Internationalisation (i18n) Setup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/issues/941"&gt;Set up vue-i18n infrastructure&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/issues/942"&gt;Create locale messages format JSON structure&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/issues/943"&gt;Allow users to change locale on the client&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;You can track the work done for these weeks through this PR:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1040"&gt;Localize browsepage and single-result page&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The progress of the project can be tracked on &lt;a href="https://github.com/creativecommons/cccatalog-frontend"&gt;cc-search&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;CC Search Accessiblity is my GSoC 2020 project under the guidance of &lt;a href="https://creativecommons.org/author/zackcreativecommons-org/"&gt;Zack Krida&lt;/a&gt; and &lt;a href="https://opensource.creativecommons.org/blog/authors/akmadian/"&gt;Ari Madian&lt;/a&gt;, who is the primary mentor for this project, &lt;a href="https://creativecommons.org/author/annacreativecommons-org/"&gt;Anna Tumadóttir&lt;/a&gt; for helping all along and engineering director &lt;a href="https://creativecommons.org/author/kriticreativecommons-org/"&gt;Kriti
Godey&lt;/a&gt;, have been very supportive.&lt;/p&gt;
</content></entry><entry><title>CC Legal Database: Coding and Mid-term status</title><link href="http://opensource.creativecommons.org/blog/entries/legal-database-coding-mid-term/" rel="alternate"></link><updated>2020-07-08T00:00:00Z</updated><author><name>krysal</name></author><id>urn:uuid:0283bfb3-3e0e-3c6a-a70b-02df13f31235</id><content type="html">&lt;p&gt;We are already in the second half of the time stipulated for the project and it is time to pause for review the initial plan, celebrate the objectives achieved and think about what remains to be done.&lt;/p&gt;
&lt;h2 id="initial-plan"&gt;Initial plan&lt;/h2&gt;&lt;p&gt;Initially, two weeks were allocated to do the redesign for the new site. I thought there would be plenty of time here, &lt;em&gt;is just design&lt;/em&gt; I said to myself, despite not having done any serious project in Figma before beyond a few sketches. Later we will see I was wrong here. This included creating new Vocabulary components if necessary. Between the second and third weeks, I would create the data models (for Django and therefore for the database as well) and from the fourth week onwards it would start to implement all this in code: make the Homepage, listing, details pages and the others.&lt;/p&gt;
&lt;h2 id="issues-in-the-way"&gt;Issues in the way&lt;/h2&gt;&lt;p&gt;One task that took longer than expected was to finish the designs, a key point because the other tasks depended on this. Though the initial scheme was ready on time, as it was discussed with the stakeholders new requirements became evident, so more modifications had to be made. For example, on the &lt;a href="https://labs.creativecommons.org/caselaw/"&gt;current site&lt;/a&gt;, the way to explore cases and scholarship is by country, and in principle, this would stay the same way and I designed with that in mind, but talking to our internal user (which acts as a &lt;em&gt;product owner&lt;/em&gt; here) was better to change this scheme to one for labels or categories that are more related with both entities. Highlighting the case of the Scholarship model, in which the attribute of the country was eliminated because it is not so relevant, and although it seemed somewhat a small thing, this also caused changes in the design of the home page, the listings and how the content of the database will be explored in general. Design for a good user experience is not so easy as a non-designer may think. There were times when there was a lack of ideas but the important thing is to make decisions and move forward, in later iterations it will be improved.&lt;/p&gt;
&lt;p&gt;As in all software development, unexpected things happen and errors will appear no matter how much you plan ahead, for the fourth week I had planned to build a continuous integration system to have a server where anyone can see the progress of my changes, however, there were a few inconveniences that had me googling for a couple of days, publishing a Django project in Heroku can be tricky, specially regarding static files (assets like style sheets and scripts) if they are generated by Heroku at some point in the deployment pipeline, depending on the phase in which it is carried out, they can be lost in the ephemeral file system of Heroku, a process that I will not delve into here but that seems important to me to highlight if anyone else  has similar problems.&lt;/p&gt;
&lt;h2 id="progress-so-far"&gt;Progress so far&lt;/h2&gt;&lt;p&gt;I have managed to finish the main tasks and I would say that even the initially expected result has been improved. So I can list the following achievements:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Redesigned the entire website using the Figma Design Library&lt;/li&gt;
&lt;li&gt;Built first pages: Home, listing and details pages for both Cases and Scholarship, and one for the FAQs.&lt;/li&gt;
&lt;li&gt;Create a GitHub Action to lint every PR and check if it follows the project's code style&lt;/li&gt;
&lt;li&gt;Deployment of the Django project on Heroku with a CI process linked to a GitHub repository, see the live development site &lt;a href="https://cc-caselaw.herokuapp.com/"&gt;here&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;It is said quickly but each task carries its considerable workload. It's been a good result so far, I've learned a lot of things along the way, like basic use of Figma, use of Storybook (related to Vocabulary components), good code security practices, some accessibility details, and more.&lt;/p&gt;
&lt;h2 id="plan-for-the-second-half-of-the-timeline"&gt;Plan for the second half of the timeline&lt;/h2&gt;&lt;p&gt;There are some tasks due from past weeks, such as build forms for Case and Scholarship submissions, but I am confident that now that the project has reached a stable state I can do it quickly in the next days. Other tasks were moved for later: searching records and filtering by tags moved after forms are created, so I can finish the visual parts of the site first and focus on functional work without shifting between types of tasks.&lt;/p&gt;
&lt;p&gt;The tasks and they order have changed, like I mentioned earlier, requirements were modified (a bit) so some tasks I planned for last weeks are not necessary anymore or are done already out-of-box with Django admin (benefits of choosing a batteries included framework!). In general, I don't think the initial plan was wrong, we just went through the natural evolution of a product software. Mentors have also been very helpful in keeping a reasonable scope and adjusting priorities.&lt;/p&gt;
&lt;p&gt;After main functionalities are done we can start making improvements, as we already identified some nice to have features but not so important at the moment. Stay tuned for more to come.&lt;/p&gt;
</content></entry><entry><title>Integration of Vocabulary with CCOS.</title><link href="http://opensource.creativecommons.org/blog/entries/integration-vocabulary-ccos/" rel="alternate"></link><updated>2020-07-08T00:00:00Z</updated><author><name>dhruvi16</name></author><id>urn:uuid:220be94d-1b9f-3d0a-8f65-fe0449aa3848</id><content type="html">&lt;p&gt;This blog demonstrates the project I am working on during the course of my Outreachy internship. My project involves redesigning and reimplementing one of the web products of the CC network using our new cohesive design library — &lt;a href="https://cc-vocabulary.netlify.app/?path=/docs/vocabulary-introduction--page"&gt;Vocabulary&lt;/a&gt;. After my internship is completed, the &lt;a href="https://opensource.creativecommons.org/"&gt;Creative Commons Open Source&lt;/a&gt; website will have a complete revamped interface that will extensively use Vocabulary components.&lt;/p&gt;
&lt;h2 id="the-problem"&gt;The Problem -&lt;/h2&gt;&lt;p&gt;There exists a large variety of web products across the Creative Commons network. These products serve different purposes for the network and so do they vary in the way of presenting the existing content. So, to conquer the inconsistency across the different products we have our coherent design system called Vocabulary.&lt;/p&gt;
&lt;p&gt;If we come to the current CC Open Source website, it is quite off-track considering our new design system Vocabulary. The styles and components of the website are not well aligned with the components of the Vocabulary. And thus it lacks harmony and consistency concerning the brand.&lt;/p&gt;
&lt;p&gt;The purpose of a design system gets dissolved if it is not well incorporated with the products and we do have this problem with the CC Open Source website. Hence, there was a need for a redesign of CCOS as per the new Design Library. This project aims to rectify the problems and inconsistencies of the CCOS website and build a completely new website (as per Vocabulary).&lt;/p&gt;
&lt;h2 id="the-solution"&gt;The Solution -&lt;/h2&gt;&lt;p&gt;I started with trying out different mock-ups for the new website in Figma. You can see what I tried to make &lt;a href="https://www.figma.com/file/ka3zs1iYnqJvyLnvAV3cW7/Home-proposals"&gt;here&lt;/a&gt;. And then with the help of our UX designer Francisco, we created all the mock-ups and finalized how the website will look in a couple of weeks, here is the &lt;a href="https://www.figma.com/file/mttcugI1UxvCJRKE5sdpbO/Mockups?node-id=759%3A516"&gt;link&lt;/a&gt; for that.&lt;/p&gt;
&lt;p&gt;I started my internship with going through the website’s tech stack, the website is made using &lt;a href="https://www.getlektor.com/"&gt;Lektor&lt;/a&gt;, with which I was not very familiar. I read the official docs, went through projects, tried making demo projects, and took help from mentors wherever I got stuck.&lt;/p&gt;
&lt;p&gt;To keep the project clean I added a theme that only has templates for the project and alters no content (as per aim). While updating templates, some components were not available in the Vocabulary yet, so I worked on adding those components to the library. And then used them in my new templates.&lt;/p&gt;
&lt;p&gt;I have thoroughly enjoyed contributing to this project as I got to learn so much. And I am excited for what’s next to come in the upcoming weeks.&lt;/p&gt;
</content></entry><entry><title>Internationalization Continued: Handling strings in the store</title><link href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week3-4/" rel="alternate"></link><updated>2020-06-26T00:00:00Z</updated><author><name>AyanChoudhary</name></author><id>urn:uuid:147fc2ec-5c3d-314b-9cf0-4baa4a84a202</id><content type="html">&lt;p&gt;These are the second two weeks of my internship with CC. I am working on improving the accessibility of cc-search and internationalizing it as well.
The internationalization work from the previous post is continued here and we also solve an interesting issue of translating strings from the Vuex store.&lt;/p&gt;
&lt;p&gt;During this period I removed all the hardcoded strings from the static pages which include:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;About Page&lt;/li&gt;
&lt;li&gt;Collections Page&lt;/li&gt;
&lt;li&gt;Search Guide Page&lt;/li&gt;
&lt;li&gt;Feedback Page &lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;All of the above pages were then internationalized following the same procedure as detailed in the previous post.
While internationalizing the homepage we ran into an interesting problem:&lt;/p&gt;
&lt;p&gt;The licenses strings were being accessed from the store and those too had to be internationalized.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;licenseTypes: [
    { code: 'commercial', name: 'Use commercially', checked: false },
    { code: 'modification', name: 'Modify or adapt', checked: false },
]
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The code was used to store the licenses in the store in the ordered format. The challenge was to extract the name strings from each license while keeping the changes and dependencies to a minimum.
The code which we were using in the templates to load these strings was:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;&amp;lt;label
        class="checkbox margin-right-big"
        :for="licenseType.code"
        :key="index"
      &amp;gt;
        &amp;lt;input
          :id="licenseType.code"
          type="checkbox"
          :checked="licenseType.checked"
          name="lt"
          :value="licenseType.code"
          @input="onFilterChanged(licenseType.code)"
        /&amp;gt;
&amp;lt;/ label&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;So after some discussion and a great insight by @brenoferreira we came up with the idea to change to name field from the store to point to our extracted internationalization strings.
This proved to be very helpful as we managed to keep changes to a minimum without using any extra dependencies.
The code after implementing the changes is:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;licenseTypes: [
    { code: 'commercial', name: 'filters.license-types.commercial', checked: false, },
    { code: 'modification', name: 'filters.license-types.modification', checked: false,},
]
&lt;/code&gt;&lt;/pre&gt;
&lt;pre&gt;&lt;code&gt;&amp;lt;label
        class="checkbox margin-right-big"
        :for="licenseType.code"
        :key="index"
      &amp;gt;
        &amp;lt;input
          :id="licenseType.code"
          type="checkbox"
          :checked="licenseType.checked"
          name="lt"
          :value="licenseType.code"
          @input="onFilterChanged(licenseType.code)"
        /&amp;gt;
        {{ $t(licenseType.name) }}
&amp;lt;/label&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The rest of the internationalization stuff was string extractions and template modifications to accomodate the the translated strings.
And we are done with these two weeks, we be back with another post after two weeks.&lt;/p&gt;
&lt;p&gt;You can track the work done for these weeks through these PRs:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1024"&gt;Internationalize About Page and Search Guide Page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1031"&gt;Internationalize feedback page, collections page and not found page&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The progress of the project can be tracked on &lt;a href="https://github.com/creativecommons/cccatalog-frontend"&gt;cc-search&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;CC Search Accessiblity is my GSoC 2020 project under the guidance of &lt;a href="https://creativecommons.org/author/zackcreativecommons-org/"&gt;Zack Krida&lt;/a&gt; and &lt;a href="https://opensource.creativecommons.org/blog/authors/akmadian/"&gt;Ari Madian&lt;/a&gt;, who is the primary mentor for this project, &lt;a href="https://creativecommons.org/author/annacreativecommons-org/"&gt;Anna Tumadóttir&lt;/a&gt; for helping all along and engineering director &lt;a href="https://creativecommons.org/author/kriticreativecommons-org/"&gt;Kriti
Godey&lt;/a&gt;, have been very supportive.&lt;/p&gt;
</content></entry><entry><title>Flickr Sub-provider Retrieval</title><link href="http://opensource.creativecommons.org/blog/entries/flickr-sub-provider-retrieval/" rel="alternate"></link><updated>2020-06-24T00:00:00Z</updated><author><name>charini</name></author><id>urn:uuid:1fcfc052-0144-3623-9d5f-f16949e32ae1</id><content type="html">&lt;h2 id="introduction"&gt;Introduction&lt;/h2&gt;&lt;p&gt;The Creative Commons (CC) licensed images made available via CC Search and CC Catalog API tools are retrieved from
numerous sources (which we refer to as providers) such as Flickr and different museum collections. While the existing
implementation of the CC Catalog tools enables filtering images in various manners such as based on image tags, the
provider, and the license type, it does not facilitate searching for images from particularly valuable internal sources
(referred to as sub-providers). For example, images related to 'NASA' have significant value in the Flickr collection,
since 'NASA' related pictures are extensively used by a large audience especially for educational purposes. The aim of
my first task in the GSoC project is to implement required changes in the API script level and in the existing data in
the database, such that filtering by certain important sub-providers is made possible.&lt;/p&gt;
&lt;p&gt;While there are several providers such as Flickr, Europeana, and Smithsonian from which we require to extract
sub-providers, the consensus was to initially focus on Flickr due to that currently being in production, and since a
substantial amount of images made available via CC Search come from Flickr. Thus, in this initial blog post, I will
discuss how I addressed the requirement of sub-provider retrieval in Flickr by making the necessary changes in the
&lt;a href="https://github.com/creativecommons/cccatalog"&gt;Creative Commons Catalog&lt;/a&gt; repository.&lt;/p&gt;
&lt;h2 id="research"&gt;Research&lt;/h2&gt;&lt;p&gt;The primary research involved in the Flickr sub-provider retrieval task was defining which entities to identify as
sub-providers, and identifying how those sub-providers can be retrieved based on the image related information we
retain.&lt;/p&gt;
&lt;h3 id="definition-of-a-sub-provider"&gt;Definition of a sub-provider&lt;/h3&gt;&lt;p&gt;It was decided that a sub-provider should be a collection of user accounts in Flickr, where this collection corresponded
to a common entity, and the common entity would reflect the sub-provider. For example, since both Flickr user accounts
&lt;em&gt;NASA HQ PHOTO&lt;/em&gt; and &lt;em&gt;NASA Johnson&lt;/em&gt; provide images related to NASA, we would represent the NASA sub-provider by those
two (and other related) user accounts.&lt;/p&gt;
&lt;p&gt;The next challenge was to determine how to identify which collections of user accounts were important to a wider
audience. The number of views per user account was an intuitive measure to rely on for this requirement. My supervisor
Brent Moran executed a query on the existing CC database to obtain the 50 most popular user accounts in Flickr. A
snippet of the query response is as follows:&lt;/p&gt;
&lt;table class="table table-striped"&gt;
&lt;thead class="thead-dark"&gt;&lt;tr&gt;
&lt;th&gt;user_account_name&lt;/th&gt;
&lt;th&gt;total_views&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Apollo Image Gallery&lt;/td&gt;
&lt;td&gt;1216297208&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BioDivLibrary&lt;/td&gt;
&lt;td&gt;625528813&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;manhhai&lt;/td&gt;
&lt;td&gt;445714729&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Thomas Hawk&lt;/td&gt;
&lt;td&gt;300554527&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sangudo&lt;/td&gt;
&lt;td&gt;258177509&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;NASA Goddard Photo and Video&lt;/td&gt;
&lt;td&gt;225143949&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Despite having a significant number of views, some of these user accounts did not appear to be worth being identified
as belonging to a sub-provider due to their lack of educational importance. Thus, we manually curated this list to
retain what we believed to be important to a wider audience.&lt;/p&gt;
&lt;h3 id="sub-provider-identification"&gt;Sub-provider identification&lt;/h3&gt;&lt;p&gt;In order to identify the sub-provider each image from Flickr belonged to, it was necessary to determine which field in
the stored image data referred to the user account. Among the various information contained in an API response, only a
selected set of fields is stored on the CC end, and it was important to use such stored data for the identification of
sub-providers. We initially decided to rely on the user account name which was reflected by the &lt;em&gt;ownername&lt;/em&gt; field in
the JSON response and stored as the &lt;em&gt;creator&lt;/em&gt; in the CC database. However, we later realised that the names of accounts
could potentially change over time, and therefore was not a reliable field for extracting the sub-provider. Another
field from the JSON response that helped to uniquely identify a user account was the &lt;em&gt;owner&lt;/em&gt; field, which acted like a
unique user ID. Even though the &lt;em&gt;owner&lt;/em&gt; value was not directly stored in the CC database, it was stored as part of the
&lt;em&gt;creator URL&lt;/em&gt; field, and fortunately, all creator URLs from Flickr consisted of a common prefix plus the &lt;em&gt;owner&lt;/em&gt; value
(the user id). Thus, we decided to use the &lt;em&gt;creator URL&lt;/em&gt; value retained in the CC database for identifying sub-providers
in Flickr.&lt;/p&gt;
&lt;h2 id="implementation"&gt;Implementation&lt;/h2&gt;&lt;p&gt;There are two levels at which sub-provider retrieval needs to be supported, where the first concerns the API scripts
from which we initially pull the data from different providers to keep the CC collections uptodate. The second is the
CC database level where the existing data needs to be updated to ensure that those reflect the sub-providers similar to
the newly added image information.&lt;/p&gt;
&lt;p&gt;The following sections explain how we represented the sub-provider information in the implementation, the changes made
at Flickr API script level and the database update logic to support sub-provider retrieval.&lt;/p&gt;
&lt;h3 id="representing-the-sub-provider-information"&gt;Representing the sub-provider information&lt;/h3&gt;&lt;p&gt;As previously explained, we define a sub-provider as a collection of user accounts, and it was identified that the
unique user ID returned in the Flickr JSON response (referred to as the &lt;em&gt;owner&lt;/em&gt;) was a reliable field for uniquely
identifying each user account. For the time being, we focused on sub-providers NASA, SpaceX, and the Biodiversity
Heritage Library (BioDivLibrary) based on their considerable importance to the community. Using the top six NASA related
user accounts, the 'Official SpaceX Photos' user account, and the 'BioDivLibrary' user account as filtered by Brent's
query, we identified the corresponding user IDs (content of the &lt;em&gt;owner&lt;/em&gt; field) using the
&lt;strong&gt;flickr.people.findByUsername&lt;/strong&gt; method made available in the Flickr API. The mapping between the sub-provider and the
corresponding user IDs was stored in a dictionary as follows.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;FLICKR_SUB_PROVIDERS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s1"&gt;&amp;#39;nasa&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s1"&gt;&amp;#39;24662369@N07&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# NASA Goddard Photo and Video&lt;/span&gt;
        &lt;span class="s1"&gt;&amp;#39;35067687@N04&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# NASA HQ PHOTO&lt;/span&gt;
        &lt;span class="s1"&gt;&amp;#39;29988733@N04&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# NASA Johnson&lt;/span&gt;
        &lt;span class="s1"&gt;&amp;#39;28634332@N05&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# NASA&amp;#39;s Marshall Space Flight Center&lt;/span&gt;
        &lt;span class="s1"&gt;&amp;#39;108488366@N07&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# NASAKennedy&lt;/span&gt;
        &lt;span class="s1"&gt;&amp;#39;136485307@N06&amp;#39;&lt;/span&gt;  &lt;span class="c1"&gt;# Apollo Image Gallery&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="s1"&gt;&amp;#39;bio_diversity&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s1"&gt;&amp;#39;61021753@N02&amp;#39;&lt;/span&gt;  &lt;span class="c1"&gt;# BioDivLibrary&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="s1"&gt;&amp;#39;spacex&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s1"&gt;&amp;#39;130608600@N05&amp;#39;&lt;/span&gt;  &lt;span class="c1"&gt;# Official SpaceX Photos&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Since this information was required both at the API script level and the database level to retrieve sub-providers, we
stored it in a common file accessible from both levels.&lt;/p&gt;
&lt;p&gt;The next challenge was to identify how to reflect the sub-provider of each image using the existing database schema.
There are two different fields in the database as &lt;em&gt;provider&lt;/em&gt; and the &lt;em&gt;source&lt;/em&gt;. The &lt;em&gt;provider&lt;/em&gt; reflects the main source
from which the images are retrieved, which happens to be 'Flickr' in this scenario. The &lt;em&gt;source&lt;/em&gt; field reflects an
organisation or entity that has published the photos using 'Flickr' in this instance (or some other site that we
recognise as a &lt;em&gt;provider&lt;/em&gt;).The &lt;em&gt;source&lt;/em&gt; field was previously not utilised and was simply set to the value of the
&lt;em&gt;provider&lt;/em&gt; in the Flickr API script. Based on internal discussions, it was decided that the &lt;em&gt;source&lt;/em&gt; field was to be
used for reflecting the sub-provider, if the corresponding image belonged to any of the user accounts contained in our
dictionary &lt;em&gt;FLICKR_SUB_PROVIDERS&lt;/em&gt;. Otherwise the &lt;em&gt;source&lt;/em&gt; was set to the default &lt;em&gt;provider&lt;/em&gt; value 'Flickr'.&lt;/p&gt;
&lt;h3 id="sub-provider-retrieval-at-api-script-level"&gt;Sub-provider retrieval at API script level&lt;/h3&gt;&lt;p&gt;Retrieving the sub-provider from the Flickr API script was fairly straightforward. Since the complete JSON response was
available at the API script level, we did not have to worry about retrieving the user ID (&lt;em&gt;owner&lt;/em&gt; value) from the
&lt;em&gt;creator URL&lt;/em&gt; field in our data. Rather, we simply get the owner value from the API response, and try to search for it
in the &lt;em&gt;FLICKR_SUB_PROVIDERS&lt;/em&gt; dictionary as follows.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;owner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;image_data&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;owner&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;next&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;FLICKR_SUB_PROVIDERS&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;owner&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;FLICKR_SUB_PROVIDERS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;]),&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;Flickr&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;Since the collection of user IDs corresponding to each sub-provider is represented as a set, the time complexity for
each sub-provider is O(1) and therefore the total time complexity is linear in the number of sub-providers (that is O(n)
for n sub-providers). Due to the number of sub-providers of interest being minimal (currently it is 3), this search
logic is quite efficient.&lt;/p&gt;
&lt;p&gt;Once we determine whether the &lt;em&gt;source&lt;/em&gt; field should be set to a sub-provider value or the default ‘Flickr’ value with
the given logic, we set the &lt;em&gt;source&lt;/em&gt; value in the image store likewise.&lt;/p&gt;
&lt;h3 id="sub-provider-update-at-the-database-level"&gt;Sub-provider update at the database level&lt;/h3&gt;&lt;p&gt;When updating sub-providers at the database level, we need to rely on the creator URL field to obtain the user ID of
each image. The creator URL is of the following form.&lt;/p&gt;
&lt;p&gt;'&lt;a href="https://www.flickr.com/photos/"&gt;https://www.flickr.com/photos/&lt;/a&gt;' + &lt;em&gt;User ID&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;For the purpose of automating the process of updating the database to reflect sub-providers, I added the necessary SQL
queries and made it accessible via the Apache Airflow UI. The database update logic is as follows.&lt;/p&gt;
&lt;p&gt;As the first step, I create a temporary table and populate it with the sub-provider values and the corresponding
creator URLs. This is done by iterating through the sub-provider, user ID value pairs in the &lt;em&gt;FLICKR_SUB_PROVIDERS&lt;/em&gt;
dictionary, and concatenating the user ID with the prefix '&lt;a href="https://www.flickr.com/photos/"&gt;https://www.flickr.com/photos/&lt;/a&gt;' to obtain the creator URL.&lt;/p&gt;
&lt;p&gt;The initial plan was to then perform a join on the CC image table (where all the image related information is stored)
with the temporary table on the condition that the creator URL from the image table matches that of the temporary table.
This query which filters all the rows in the image table where we need to update the sub-provider values, looks as
follows.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;UPDATE&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;image_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;SET&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SOURCE&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;public&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;temp_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SUB_PROVIDER&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;public&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;temp_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;WHERE&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;image_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CREATOR_URL&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;public&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;temp_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CREATOR_URL&lt;/span&gt;&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;However, a major concern with this query, as my supervisor Brent Moran pointed out, was that it locked all the rows
which matched the 'WHERE' clause at once. With respect to the magnitude of the Flickr data available in the CC image
table, this meant that the above query would lock millions of rows, thus hindering the execution of other queries on
the image table. To mitigate this issue, we decided to update the SQL query as follows, such that we perform a 'SELECT'
query on the rows to be updated by joining the previously created temporary table with the CC image table (a 'SELECT'
query does not lock the table), and iterate row by row over the query result to set the &lt;em&gt;source&lt;/em&gt; value in the image
table to the sub-provider value.&lt;/p&gt;
&lt;div class="hll"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;span class="n"&gt;SELECT&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FOREIGN_ID&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="n"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;foreign_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="n"&gt;public&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;temp_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;PROVIDER&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="n"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;sub_provider&lt;/span&gt;
&lt;span class="n"&gt;FROM&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;image_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;INNER&lt;/span&gt; &lt;span class="n"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;public&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;temp_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;ON&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;image_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CREATOR_URL&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;public&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;temp_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CREATOR_URL&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;AND&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;image_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;PROVIDER&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;Flickr&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;# Let us refer to the result produced from the SELECT query as &amp;#39;selected_records&amp;#39;&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;foreign_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sub_provider&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;selected_records&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;UPDATE&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;image_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;SET&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;SOURCE&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;{sub_provider}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
    &lt;span class="n"&gt;WHERE&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;image_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;PROVIDER&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;Flickr&amp;#39;&lt;/span&gt;
    &lt;span class="n"&gt;AND&lt;/span&gt;
    &lt;span class="n"&gt;MD5&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="n"&gt;image_table&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;col&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FOREIGN_ID&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;MD5&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;{foreign_id}&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/pre&gt;&lt;/div&gt;
&lt;p&gt;To make this functionality available from the Airflow UI, I have added the Airflow DAG
&lt;em&gt;flickr_sub_provider_update_workflow&lt;/em&gt;.
The changes in the source field after updating the image table in the database looks like follows.&lt;/p&gt;
&lt;table class="table table-striped"&gt;
&lt;thead class="thead-dark"&gt;&lt;tr&gt;
&lt;th style="text-align:center"&gt;id&lt;/th&gt;
&lt;th style="text-align:center"&gt;provider&lt;/th&gt;
&lt;th style="text-align:center"&gt;source before the update&lt;/th&gt;
&lt;th style="text-align:center"&gt;source after the update&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;14369&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;bio_diversity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;14372&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;bio_diversity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;14375&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;bio_diversity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;14378&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;bio_diversity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;14382&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;bio_diversity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;40784&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;nasa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;47237&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;nasa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;47242&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;nasa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;47244&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;nasa&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td style="text-align:center"&gt;47245&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;flickr&lt;/td&gt;
&lt;td style="text-align:center"&gt;nasa&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;For more information regarding the implementation, please refer to the following PR:
&lt;a href="https://github.com/creativecommons/cccatalog/pull/420"&gt;https://github.com/creativecommons/cccatalog/pull/420&lt;/a&gt;&lt;/p&gt;
&lt;h2 id="acknowledgement"&gt;Acknowledgement&lt;/h2&gt;&lt;p&gt;I express my gratitude to Brent Moran and Anna Tumadóttir for their assistance with my first task in GSoC 2020 by
helping me to filter the sub-providers of interest and conducting the necessary research.&lt;/p&gt;
</content></entry><entry><title>CC Search, Setting up vue-i18n and internationalizing homepage</title><link href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-week1-2/" rel="alternate"></link><updated>2020-06-10T00:00:00Z</updated><author><name>AyanChoudhary</name></author><id>urn:uuid:b6330fc6-4ba0-3d32-96cf-057eec67072a</id><content type="html">&lt;p&gt;These are the first two weeks of my internship with CC. I am working on improving the accessibility of cc-search and internationalizing it as well.
We started with first compiling the accessibility reports from accessibility insights, lighthouse and pa11y into a single document and then opening up appropriate issues ont he repo to address them.&lt;/p&gt;
&lt;p&gt;The accessibility issues are listed here:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/issues/996"&gt;Accessibility - Improve labels&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/issues/997"&gt;Evaluate keyboard navigation effectiveness&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/issues/998"&gt;Fix color contrast problems&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/issues/999"&gt;Improve elements markup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/issues/1000"&gt;Evaluate any accessibility linter tools&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The decision was made to audit the tab indices along with internationalizing the page.
The accessibility changes will be done after the completion of internationalization as the aria-labels will have to be internationalized as well.&lt;/p&gt;
&lt;p&gt;The first two weeks involved setting up vue-i18n, auditing the tab index for homepage and internationalizing it.
The tab index adit for homepage is displayed:&lt;/p&gt;
&lt;p&gt;&lt;img src="/blog/entries/cc-search-accessibility-week1-2/audit.png" alt="audit.png"&gt;&lt;/p&gt;
&lt;p&gt;The internationalization part was pretty straightforward, we just had to export all the strings to the JSON files and load transaltions through the i18n module.
For complex elements of the type &lt;code&gt;string &amp;lt;tag&amp;gt;string&amp;lt;/tag&amp;gt; string&lt;/code&gt; I went for the templating method.
Here we use the v-slot attribute of the i18n functional component to convert the element into a template where the tag occupies a slot in the syntax.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;&amp;lt;i18n path="footer.caption.label" tag="p" class="caption"&amp;gt;
    &amp;lt;template v-slot:noted&amp;gt;
        &amp;lt;a href="https://creativecommons.org/policies#license" target="_blank" rel="noopener"&amp;gt;{{$t('footer.caption.noted')}}&amp;lt;/a&amp;gt;
    &amp;lt;/template&amp;gt;
    &amp;lt;template v-slot:attribution&amp;gt;
        &amp;lt;a href="https://creativecommons.org/licenses/by/4.0/" target="_blank" rel="noopener"&amp;gt;
        {{$t('footer.caption.attribution')}}
        &amp;lt;/a&amp;gt;
    &amp;lt;/template&amp;gt;
    &amp;lt;template v-slot:icons&amp;gt;
        &amp;lt;a href="https://fontawesome.com/" target="_blank" rel="noopener" class="has-text-white"&amp;gt;
        {{$t('footer.caption.icons')}}
        &amp;lt;/a&amp;gt;
    &amp;lt;/template&amp;gt;
&amp;lt;/i18n&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The final outcome looks pretty good:&lt;/p&gt;
&lt;p&gt;&lt;img src="/blog/entries/cc-search-accessibility-week1-2/final.png" alt="final.png"&gt;&lt;/p&gt;
&lt;p&gt;And voila we are done with the first two weeks. I also internationalized the header and the footer along with the homepage.
You can track the work done for these weeks through these PRs:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1007"&gt;setup internationalization plugin&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog-frontend/pull/1013"&gt;Internationalize homepage, header and footer&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The progress of the project can be tracked on &lt;a href="https://github.com/creativecommons/cccatalog-frontend"&gt;cc-search&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;CC Search Accessiblity is my GSoC 2020 project under the guidance of &lt;a href="https://opensource.creativecommons.org/blog/authors/akmadian/"&gt;Ari Madian&lt;/a&gt;, who is the primary mentor for this project, &lt;a href="https://creativecommons.org/author/annacreativecommons-org/"&gt;Anna Tumadóttir&lt;/a&gt; for helping all along and engineering director &lt;a href="https://creativecommons.org/author/kriticreativecommons-org/"&gt;Kriti
Godey&lt;/a&gt;, have been very supportive.&lt;/p&gt;
</content></entry><entry><title>Science Museum provider implementation</title><link href="http://opensource.creativecommons.org/blog/entries/science-museum-implementation/" rel="alternate"></link><updated>2020-06-10T00:00:00Z</updated><author><name>srinidhi</name></author><id>urn:uuid:7960ba6c-0bbb-300e-86d8-4744b9b0ee25</id><content type="html">&lt;h2 id="introduction"&gt;Introduction&lt;/h2&gt;&lt;p&gt;CC catalog project is responsible for collecting CC licensed images available in the web,  CC licensed images are hosted by different
sources, these sources that provide the images and its metadata are called providers. Currently, images are collected from providers using two methods
Common Crawl and API based crawl. Common Crawl data is an open repository of web crawled data and we use that data to get the necessary image metadata 
for that provider &lt;a href="https://commoncrawl.org/the-data/get-started/"&gt;more information&lt;/a&gt;. API crawl is implemented using the API endpoint maintained 
by the providers. The main problem with Common Crawl is that we don't have control over the data they crawl, and this sometimes results poor 
data quality whereas with API based crawl we have access to the information available. API based crawl is better when we need to update image
metadata and reqular intervals.&lt;/p&gt;
&lt;p&gt;As a part of the internship, I will be working on moving providers from Common Crawl to API based crawl as well as integrate new providers
to the API crawl. I will be starting with the Science Museum provider.&lt;/p&gt;
&lt;h2 id="science-museum"&gt;Science Museum&lt;/h2&gt;&lt;p&gt;Science museum is a provider with around 80,000 CC licensed images, currently Science museum data is ingested from Common Crawl. 
Science museum is one such provider where our data is of poor quality and there is need to improve it. This is done by moving 
Science museum to an API based crawl.&lt;/p&gt;
&lt;h2 id="api-research"&gt;API research&lt;/h2&gt;&lt;p&gt;We want to index metadata using their open API &lt;a href="https://collection.sciencemuseumgroup.org.uk/search/has_image/image_license"&gt;endpoint&lt;/a&gt;. 
However, before the implementation we have to ensure that the API provides necessary content and there is a systematic way to get it.
The first step is to take an object from their collection and check certain criterias.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://collection.sciencemuseumgroup.org.uk/api/objects/co8005638"&gt;sample object&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;The criteria are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;parameters available for the API&lt;/li&gt;
&lt;li&gt;Object landing url (frontend link of the object the image is associated with) &lt;/li&gt;
&lt;li&gt;Image url (the url link of the image)&lt;/li&gt;
&lt;li&gt;CC license associated with the image&lt;/li&gt;
&lt;li&gt;creator, title and other metadata info &lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Once the above checks have been made, we need to find a way to get all the objects, this could be by paging through the records 
or partition using the parameters, etc. Since their API parameter has &lt;code&gt;page[number]&lt;/code&gt; paging would be an appropriate choice with  max size 
as 100 it would require around 800 pages to get all the objects but then since they don't allow paging a large number of results, and 
the max number of pages for Science Museum is 50 pages.This would mean we would get only 5000 objects and around 17000 images.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://collection.sciencemuseumgroup.org.uk/search/image_license?page[size]=100&amp;amp;page[number]=50"&gt;API page-50&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href="https://collection.sciencemuseumgroup.org.uk/search/image_license?page[size]=100&amp;amp;page[number]=51"&gt;API page-51&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;So we need to find a way to divide the collection into subsets such that each subset has less than or equal to 5000 objects.
Luckily, the API had another set of parameters &lt;code&gt;date[from]&lt;/code&gt; and &lt;code&gt;date[to]&lt;/code&gt; which represents the time period of the object. 
Querying the API through different time period at the same time ensuring that records in that time period don't exceed 5000 solves the problem, starting
from year 0 to year 2020 by trial and error method suitable year range was chosen.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;                                            YEAR_RANGE = [
                                                (0, 1500),
                                                (1500, 1750),
                                                (1750, 1825),
                                                (1825, 1850),
                                                (1850, 1875),
                                                (1875, 1900),
                                                (1900, 1915),
                                                (1915, 1940),
                                                (1940, 1965),
                                                (1965, 1990),
                                                (1990, 2020)
                                            ]
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;With this we have a method to ingest the desired records, but before writing the script we need to know the different licenses 
provided by the API.  We need to figure out a consistent way to identify which license and version are attached to each object.&lt;br&gt;
To do this, we ran a test script to get counts of objects under different licenses.&lt;/p&gt;
&lt;p&gt;The results are:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;+-----------------+----------+
| license_version | count(1) |
+-----------------+----------+
| CC-BY-NC-ND 2.0 |      210 |
| CC-BY-NC-ND 4.0 |     2376 |
| CC-BY-NC-SA 2.0 |        1 |
| CC-BY-NC-SA 4.0 |    61694 |
+-----------------+----------+
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Since the licenses and their versions are confirmed, we can start the implementation.&lt;/p&gt;
&lt;h2 id="implementation"&gt;Implementation&lt;/h2&gt;&lt;p&gt;The implementation is quite simple in nature: we loop the through the &lt;code&gt;YEAR_RANGE&lt;/code&gt; and get all the records for that period and 
pass it on to an object data handler method that extracts the necessary details from the record and store it in the &lt;code&gt;ImageStore&lt;/code&gt;
instance. ImageStore is a class that stores image information from the provider, it stores the information in a buffer and inserts to tsv
when the buffer reached threshold limit. Due to overlapping date ranges, the metadata for some objects is collected multiple times.
So, we keep track of the record/object's id in a global variable &lt;code&gt;RECORD_IDS = []&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Within the object data handler method before collecting details we check if the &lt;code&gt;id&lt;/code&gt; already exists in &lt;code&gt;RECORD_IDS&lt;/code&gt;. 
If it exists we move on to the next record.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;                                            for obj_ in batch_data:
                                                id_ = obj_.get("id")
                                                if id_ in RECORD_IDS:
                                                    continue
                                                RECORD_IDS.append(id_)
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;code&gt;id_&lt;/code&gt; is the object id and we cannnot use this value as foreign identifier, the reason behind it is that an object could
have multiple images with it and using object id we cannot determine the image uniquely, so we must use image id that is unique
for each image. Currently image id is taken from &lt;code&gt;multimedia&lt;/code&gt;, multimedia is a field in the json response that lists multiple 
images and their metadata, for each image data in multimedia, foreign id is in &lt;code&gt;admin.uid&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The implementation can be found &lt;a href="https://github.com/creativecommons/cccatalog/blob/master/src/cc_catalog_airflow/dags/provider_api_scripts/science_museum.py"&gt;here&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id="results"&gt;Results:&lt;/h3&gt;&lt;p&gt;Running the scripts we get:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Number of records recieved : &lt;code&gt;35584&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Number of images collected : &lt;code&gt;62497&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The problem with current implementation is that records with no date would be missed.&lt;/p&gt;
&lt;p&gt;Science Museum provider is the first provider I worked on as a part of the internship and thank my mentor Brent Moran for the help.&lt;/p&gt;
&lt;h3 id="additional-details"&gt;Additional Details :&lt;/h3&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog/issues/302"&gt;research work&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/creativecommons/cccatalog/pull/400"&gt;implementation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
</content></entry><entry><title>CC Legal Database: Design</title><link href="http://opensource.creativecommons.org/blog/entries/legal-database-design/" rel="alternate"></link><updated>2020-06-09T00:00:00Z</updated><author><name>krysal</name></author><id>urn:uuid:b2a2a092-d722-34fb-bb65-9103c9b4ed48</id><content type="html">&lt;p&gt;Finishing the third week since the project started (for context see this &lt;a href="/blog/entries/legal-database-a-new-beginning/"&gt;first post&lt;/a&gt;), so the design phase is almost over and a new site look is out of the oven. The focus on these weeks was to draw the mockups for the user-facing parts of the site, integrating styles of CC Vocabulary and to get the data model for the database.&lt;/p&gt;
&lt;h3 id="visual-styles"&gt;Visual Styles&lt;/h3&gt;&lt;p&gt;The intention was to keep the content that is already present but improve its distribution and access by users. For this, the main menu was changed to provide direct links to listing of Cases and Scholarships. The old "Countries" page was removed and replaced by a more granular division by legal resource, so this data will be shown separately.&lt;/p&gt;
&lt;p&gt;The final look for the home site is as follows.&lt;/p&gt;
&lt;div style="text-align: center;"&gt;
    &lt;figure&gt;
        &lt;img src="cc-caselaw-home.png" alt="New CC Caselaw Home Mockup" style="border: 1px solid black"&gt;
        &lt;figcaption&gt;New Home page design with Vocabulary.&lt;/figcaption&gt;
    &lt;/figure&gt;
&lt;/div&gt;&lt;p&gt;I made use of as many Vocabulary components as possible, like header, footer and table. This way is easier to keep consistency between CC products and to develop the frontend part of the site because those components are already built and tested, though some will require certain modifications (e.g. card link with a search input), and some others have to be created from scratch, like a pagination component that is actually now required for two sites.&lt;/p&gt;
&lt;h3 id="data-model"&gt;Data Model&lt;/h3&gt;&lt;p&gt;The second main task I worked on was coding the models on Django, which is in charge of creating the database schema through migrations. For this, I had to review the sources of information (CSV files, sheets, forms) and how they are used. The key point here is to keep constant communication with staff who are more involved in the &lt;em&gt;business case&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Several iterations were required for each task as well as some researching, and while the engineering and design work never seems to end, this makes good foundations to continue and advance.&lt;/p&gt;
</content></entry><entry><title>Resource Gathering</title><link href="http://opensource.creativecommons.org/blog/entries/resource-gathering/" rel="alternate"></link><updated>2020-06-09T00:00:00Z</updated><author><name>dhruvi16</name></author><id>urn:uuid:1b4acbcb-fbab-372c-854f-833a05aafa23</id><content type="html">&lt;p&gt;As an outreachy intern, I am handling the integration of &lt;a href="https://creativecommons.org/"&gt;Creative Commons&lt;/a&gt; design library — &lt;a href="https://cc-vocabulary.netlify.app/"&gt;Vocabulary&lt;/a&gt; with one of our web products — &lt;a href="https://opensource.creativecommons.org/"&gt;CC OS&lt;/a&gt;. I have been working the design library for 3–4 months now and I have enjoyed the experience the library caters and I am trying to achieve the same experience in the Open source website. To understand UX in-depth, I have been reading different resources and document this knowledge through this series of blogs. This in-depth information will help me achieve the desired experience through the library.&lt;/p&gt;
&lt;p&gt;Using a Coursera course, &lt;a href="https://www.coursera.org/learn/user-experience-design/"&gt;Introduction to user experience&lt;/a&gt;, I will be describing the UX design cycle with a series of articles and this article is about describing the first step of the design cycle which is Resource Gathering.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Basic definitions&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;User experience design includes designing interfaces through which a user accomplishes a task. Designing better interfaces which can help the user to perform tasks easily.&lt;/p&gt;
&lt;p&gt;The interface consists of an input and output through which the user interacts with the system. For instance, clicking a photo requires the user to press the button (input) and an image is the desired output. Creating an affordable and usable interface is the main goal of this process. Design is a data-driven process and resource gathering is all about gathering this data.&lt;/p&gt;
&lt;p&gt;The resource gathering process is about figuring out how the task is currently accomplished by the user. There are 4 ways to gather data and below I will describe them all in detail. There are two types of data — Quantitative (numeric) and Qualitative (thematic) and designers prefer to use both types of data as per requirement.&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Naturalistic observation&lt;/strong&gt; - This includes observing the user accomplishing the task in the field. This involves the least interaction with the user and the designer watches the user performing the task from distance. The designer notes down qualitative and quantitative information about this activity. This removes the effect of social desirability of the user on the information collected but also the designer’s perception can be reflected in the collected data.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Surveys&lt;/strong&gt; - A survey can be interchangeably used with a questionnaire. In a survey, the user answers a set of questions about how he/she performs the tasks currently. The questions can be closed-ended which can provide quantitative data and also open-ended which gives us the qualitative data. This involves some amount of interaction with the user. Surveys can be held in the field or lab.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Focus Groups&lt;/strong&gt; - Focus groups are about engaging with a group of 6–10 people and talk about how they perform a task currently. This involves a lot of interaction with the users. This can be performed in a safe environment (lab) where users can open up without hesitation. The design team includes a moderator who can ask relevant questions, a note-taker who can note down the on-going conversation and a media person (optional) who can record video or take photos of the session.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Interview&lt;/strong&gt; - The interview involves asking questions to the user one-to-one about how they perform the task currently. This involves the highest amount of interaction with the user. Interviews are held in labs. The designer talks to the user about the task and collects both quantitative and qualitative data. This is the most time-taking way of collecting data but it gives the most useful data among all the methods.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
</content></entry><entry><title>Upcoming improvements/features in CC Search Extension</title><link href="http://opensource.creativecommons.org/blog/entries/cc-browser-extension-upcoming-improvements/" rel="alternate"></link><updated>2020-06-01T00:00:00Z</updated><author><name>makkoncept</name></author><id>urn:uuid:cdf59a68-68ac-36b4-85c8-a19bb2848f33</id><content type="html">&lt;p&gt;&lt;a href="https://opensource.creativecommons.org/ccsearch-browser-extension/"&gt;CC Search Extension&lt;/a&gt; is a cross-browser extension, which lets you search for and filter content that is under Creative Commons licenses. It was developed as one of the CC projects during Google Summer of Code program of 2019.&lt;/p&gt;
&lt;h3 id="the-story-so-far"&gt;The story so far&lt;/h3&gt;&lt;p&gt;It's release on the extension stores of &lt;a href="https://chrome.google.com/webstore/detail/cc-search-browser-extensi/agohkbfananbebiaphblgcfhcclklfnh"&gt;Chrome&lt;/a&gt;, &lt;a href="https://addons.mozilla.org/en-US/firefox/addon/cc-search-extension/"&gt;Firefox&lt;/a&gt;, and &lt;a href="https://addons.opera.com/en/extensions/details/cc-search/"&gt;Opera&lt;/a&gt; was accompanied by several announcements on twitter, which were well received by the community and thus the number of weekly users (i.e the number of users that have used the extension at least once during last week) drastically increased. After several weeks, the extension reached 22,000+ weekly users. But, now this number fluctuates between 9,000 - 10,000. So, what happened? Well, the majority of initial influx were the "curious" folks who just wanted to check out this new tool and later thought that this was not something that they might find useful. But, there are also users that are looking for a similar tool and the lack of sufficient features and capabilities of the extension made them stop using it and look for better alternatives.&lt;/p&gt;
&lt;h3 id="what-s-the-plan"&gt;What's the plan&lt;/h3&gt;&lt;p&gt;The plan is to transform the extension to become competent enough to contribute to the user's workflow and be an important part of their toolkit. This will be done by adding valuable new features in addition to other enhancements and improvements.&lt;/p&gt;
&lt;p&gt;In this section, I will list several improvements and features that will be added to the extension during the upcoming months.&lt;/p&gt;
&lt;h4 id="new-filters"&gt;New Filters&lt;/h4&gt;&lt;p&gt;The extension, currently supports only 3 filters: &lt;code&gt;license&lt;/code&gt;, &lt;code&gt;sources&lt;/code&gt;, and &lt;code&gt;use case&lt;/code&gt;. The support for new filters like &lt;code&gt;image type&lt;/code&gt;, &lt;code&gt;file type&lt;/code&gt;, &lt;code&gt;aspect ratio&lt;/code&gt;,  and &lt;code&gt;image size&lt;/code&gt; will be added. This will give more options to the users and let them be more specific about the content they want.&lt;/p&gt;
&lt;h4 id="related-images-and-image-tags"&gt;Related Images and Image tags&lt;/h4&gt;&lt;p&gt;Showing related Images will help users find a variety of images that fit their requirements and also explore the images that would not usally show up on the initial pages of the search result. Whereas, Image tags will let the users incrementally make their queries better and more specific.&lt;/p&gt;
&lt;h4 id="adding-browse-by-sources-section"&gt;Adding "Browse by Sources" section&lt;/h4&gt;&lt;p&gt;Even though the users can see the number of sources supported by the extension from the drop-down filter, they might not be familiar with many of them and what kind of content they provide. Adding a "Browse by Source" section would make users appreciate the sources available for them to choose from.&lt;/p&gt;
&lt;h4 id="improving-the-bookmarks-section"&gt;Improving the bookmarks section&lt;/h4&gt;&lt;p&gt;Significant improvements will be made to the bookmarks section. First of all, the bookmarks data will be cached so that unnecessary requests to the Catalog API can be avoided. The bookmarks will be organized by the dates and this will also facilitate adding some type of filter mechanism. In addition to this, pagination in the bookmarks section and support for named bookmark file exports will be added&lt;/p&gt;
&lt;h4 id="syncing-as-much-data-as-possible"&gt;Syncing as much data as possible&lt;/h4&gt;&lt;p&gt;An effort will be made to sync as much extension data as possible between the user's system. Syncing the user's preference/settings will ensure a familiar experience across systems and syncing the bookmarks data and the previous session search results would make them feel like they are continuing from where they left.&lt;/p&gt;
&lt;h4 id="integrating-cc-vocabulary"&gt;Integrating CC Vocabulary&lt;/h4&gt;&lt;p&gt;The extension uses a very outdated version of CC Vocabulary. It will be replaced by the latest version for a refreshing new look.&lt;/p&gt;
&lt;h4 id="adding-support-for-microsoft-edge"&gt;Adding support for Microsoft Edge&lt;/h4&gt;&lt;p&gt;The new Microsoft Edge browser is based on chromium. This paves the way to add support for it thanks to the power and flexibility of the WebExtension API. This will then allow us to also officially release on the &lt;a href="https://microsoftedge.microsoft.com/addons/Microsoft-Edge-Extensions-Home"&gt;Microsoft Edge Addon Store&lt;/a&gt;.&lt;/p&gt;
&lt;h3 id="join-the-discussion"&gt;Join the discussion&lt;/h3&gt;&lt;p&gt;If you have any feedback/suggestions or just want to say hi,  please join the &lt;code&gt;#cc-dev-browser-extension&lt;/code&gt; channel on &lt;a href="https://opensource.creativecommons.org/community/#slack"&gt;slack&lt;/a&gt;. You can also track the progress on the project's &lt;a href="https://github.com/creativecommons/ccsearch-browser-extension"&gt;Github&lt;/a&gt; Repository.&lt;/p&gt;
</content></entry><entry><title>CC Search, Proposal Drafting and Community Bonding</title><link href="http://opensource.creativecommons.org/blog/entries/cc-search-accessibility-and-internationalization/" rel="alternate"></link><updated>2020-05-22T00:00:00Z</updated><author><name>AyanChoudhary</name></author><id>urn:uuid:e9ce22b5-b66a-37f1-a59b-aa5419b7d5c5</id><content type="html">&lt;h3 id="proposal-drafting"&gt;Proposal Drafting&lt;/h3&gt;&lt;p&gt;The majority of my time in March was spent on drafting the proposal for my project &lt;strong&gt;Improve CC Search Accessibility&lt;/strong&gt;.
While drafting my proposal I had two broad topic that I had to focus on: Accessibility and Internationalization.&lt;/p&gt;
&lt;p&gt;So the first thing which I did was go through the various resources available with me such as w3 guidelines for accessibility, dequeuniversity accessibility insights and MDN notes on accessibility.
After I made myself acquainted myself wih all of these, the next challenge was to sort out which of the metrics were relevant and important enough to be detailed in the proposal and also some of the others metrics which made notable appearances.
Finally by including all of these I had the accessibility part of my proposal complete. Next, I had to work out the part for internationalization. Since it was already decided upon that we will be using vue-i18n, I did some research as to how to we can leverage it to gain the best possible result.&lt;/p&gt;
&lt;p&gt;One of the important parts of internationalization happens to be deciding upon the JSON structure which was a highlighted section in my proposal.
The other notable sections included strategies for modification of templates while translating and also how the translations would be carried out without hindering any further development of the platform.&lt;/p&gt;
&lt;h3 id="community-bonding"&gt;Community Bonding&lt;/h3&gt;&lt;p&gt;Community Bonding involved getting to the mentors and the people whom I will be working with during this internship. Also we decided upon running the audit tests for the cc-search website during this time as it would help identify the key issues we would be facing and also would provide a suitable foundation to start working upon.
The audits were done using Lighthouse, Accessibility Insights and pa11y and they provided useful insights on which parts of the website we should be focusing on such as the contrast issues and the aria-label fixes.&lt;/p&gt;
&lt;p&gt;Coming up next will be the progress on the first 2 weeks of the project.&lt;/p&gt;
</content></entry><entry><title>Takeaways from my FOSS journey!</title><link href="http://opensource.creativecommons.org/blog/entries/takeaways-from-my-FOSS-journey/" rel="alternate"></link><updated>2020-05-22T00:00:00Z</updated><author><name>dhruvi16</name></author><id>urn:uuid:21536df2-1436-3e06-8262-8e5d2c91e01c</id><content type="html">&lt;div style="text-align: center;"&gt;
  &lt;figure&gt;
    &lt;img src="cover.jpg"/&gt;
    &lt;figcaption&gt;
      &lt;em&gt;
        &lt;a href="https://www.flickr.com/photos/60255995@N04/14774214687"&gt;"Sky at night"&lt;/a&gt; by &lt;a href="https://www.flickr.com/photos/60255995@N04"&gt;Rychu92&lt;/a&gt; is licensed under &lt;a href="https://creativecommons.org/licenses/by-sa/2.0/?ref=ccsearch&amp;atype=rich"&gt;CC BY-SA 2.0&lt;/a&gt;
      &lt;/em&gt;
    &lt;/figcaption&gt;
  &lt;/figure&gt;
&lt;/div&gt;&lt;p&gt;First let me introduce myself because of course I am not a very famous blogger or writer.&lt;/p&gt;
&lt;p&gt;I am Dhruvi. I am a third-year undergrad student. I am a designer/developer and also a FOSS enthusiast. I started my opensource journey a year ago (that is around Feb ’19) and now here I am starting my Outreachy internship with this blog (not literally of course). I am consistent, curious, and creative, and below I will tell you how these three C’s have helped me through my journey.&lt;/p&gt;
&lt;p&gt;Below are the four things that I think might be the ‘takeaways’ from my experience -&lt;/p&gt;
&lt;h3 id="now-is-the-time"&gt;NOW is the time -&lt;/h3&gt;&lt;p&gt;There is no better time than now! Let’s accept the fact that most of the students wait for “the time” to start with opensource as most of us want to get into programs like GSoC, Outreachy, and many more. But I figured it is very necessary to know that whether you are interested in opensource or is it only the big internships attracting you. So, to get these answers, you need to explore, and to explore I suggest you start today. I am very fond of asking myself the question ‘Why?’ and to answer those questions I explore, I invest time. So start finding answers to your whys? today.&lt;/p&gt;
&lt;h3 id="start-slow"&gt;Start SLOW -&lt;/h3&gt;&lt;p&gt;For me the start was a little overwhelming and that made me push it away. And to avoid that I tried starting small and slow. By small and slow, I mean try the easy things first, like getting aware of the logistics. Personally, general practices or set of rules calm me down when I try to get into some uncomfortable zones. And yes opensource was not a very comfortable zone for me in the beginning (I mean what is *wink). To keep myself motivated, I started with familiar things like contributing to familiar projects or familiar good first issues (the easy ones). Completing a task or two gave me a lot of confidence and made me stick around.&lt;/p&gt;
&lt;h3 id="choose-wisely"&gt;Choose WISELY -&lt;/h3&gt;&lt;p&gt;After playing around for some time and getting familiar with the logistics, I went for a project hunt. I listed my areas of interest and areas which seemed exciting to me. Using those filters I found some amazing projects which were actively being worked on. I got my answers to why would I want to work on these projects? by working on them. I believe if you are interested the motivation will be natural. With some projects, I felt connected and I did not feel the need of pushing myself for contributing to them. My only advice for making a wise choice is that “Don’t push yourself too hard”!&lt;/p&gt;
&lt;h3 id="speak-through-your-work"&gt;Speak through your WORK -&lt;/h3&gt;&lt;p&gt;When I was sure, I wanted to work on a particular project I started getting familiar with the community (the specifics). I got familiar with the ideas and agendas the community had behind the projects. And fortunately, I got to work with the most amazing and inspiring people who helped me when I was stuck and I plan to keep working with them. I kept doing a few things which I guess might be worth mentioning -&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Try to write the most appropriate and relevant code.&lt;/li&gt;
&lt;li&gt;Try to break the solution into small understandable parts.&lt;/li&gt;
&lt;li&gt;Try to take initiatives. Small initiatives like opening issues will also work (Yes! opening relevant and meaningful issues is an initiative.).&lt;/li&gt;
&lt;li&gt;Know when to ask for help from the community (I am still working on this).&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;I know this was not an article full of resources and tips to get into Outreachy or GSoC but I just wanted to communicate that students should see that opensource is a broader concept. I wanted to share a personal take and help the beginners with my own experience. Thanks for reading!&lt;/p&gt;
</content></entry><entry><title>CC Legal Database: a new beginning</title><link href="http://opensource.creativecommons.org/blog/entries/legal-database-a-new-beginning/" rel="alternate"></link><updated>2020-05-22T00:00:00Z</updated><author><name>krysal</name></author><id>urn:uuid:0ca867d0-f061-3202-9da4-7e89977781ce</id><content type="html">&lt;p&gt;CC maintains a collection of case law and legal scholarship relevant to legal issues around Creative Commons licenses. The CC Legal Database system gathers and exposes this information for everybody, as anyone can submit information, this aims to help understanding how CC legal tools have been interpreted by courts, and how they operate the larger legal ecosystem.&lt;/p&gt;
&lt;p&gt;The current &lt;a href="https://labs.creativecommons.org/caselaw/"&gt;site&lt;/a&gt; is a beta version built on Jekyll, a static site generator, using Google forms and spreadsheets as database. This system has served its purpose, but it's time to modernize it since there are several issues we want to fix, mainly related to the automation of review and publication process, as for the visual styles, CC is doing a major redesign of websites with &lt;a href="https://cc-vocabulary.netlify.com/"&gt;Vocabulary&lt;/a&gt;, the new design system, to unify the user experience.&lt;/p&gt;
&lt;p&gt;The process for collecting the information is decoupled into loose spreadsheets, some Google forms, and its publication is a cumbersome manual process for two teams involved, the Legal Team that have to review the incoming data and edit it if necessary, and the Tech Team which is the one that actually publishes. This is prone to human errors (e.g. missing a column entry for one row) that can be easily avoided as well as save time to both CC teams.&lt;/p&gt;
&lt;p&gt;This project, Reimplement CC’s Legal Database using Django, aims to address all these main issues, automate the reviewing and publication process, empowering CC legal staff to manage all the workflow without the intervention of the tech team, and make use of Vocabulary components and guides to unify styles with the rest of Creative Commons products.&lt;/p&gt;
&lt;h3 id="design"&gt;Design&lt;/h3&gt;&lt;p&gt;First task in the plan is to design the user-facing parts of the new website on Figma, this tool allows to easily draw on the browser (great for start quickly since no need to install a program in your PC) and even collaborate between peers to edit one document at the same time. CC keeps here the Vocabulary design library, so it facilitates a lot using the already existing components.&lt;/p&gt;
&lt;p&gt;You can find designs and follow the process in this Figma link: &lt;a href="https://www.figma.com/file/L2sACqzX61N2OMd5lmisop/New-CC-Caselaw-Mockups"&gt;New CC Caselaw Mockups&lt;/a&gt;. A new logo was made by the UX designer Francisco, with this the project start feeling like a consistent product of CC.&lt;/p&gt;
&lt;div style="text-align: center; margin: 15px 0px"&gt;
&lt;figure&gt;
    &lt;img src="legal_database.svg" alt="New Creative Commons Legal Database logo" width="350px"/&gt;
&lt;/figure&gt;
&lt;/div&gt;&lt;!-- ![New Creative Commons Legal Database logo](legal_database.svg) --&gt;

&lt;h3 id="why-django"&gt;Why Django?&lt;/h3&gt;&lt;p&gt;Before I briefly mentioned the use of Django, a framework of Python, for the implementation, so I will state some reasons that led to this decision.&lt;/p&gt;
&lt;p&gt;Two options were given for the backend development: WordPress or Django. WordPress is a nice tool to build sites like blogs and e-commerce, it gives a lot of functionalities to end-users (usually not technical) out of the box and can be extended with a wide ecosystem of plugins, but I identified some cons for this particular case. It will bring many not necessary features to the system initially planned and will take considerable work to customize both, the frontend with Vocabulary and the database in the backend. Building a Wordpress theme with Vocabulary was already considered for another project apart, so that should be done first. As for the DB,  deal with Wordpress tables may not be the best developer experience and could be a real mess.&lt;/p&gt;
&lt;p&gt;Looking at Django with Python, we can start in a greenfield ready to customize, like Wordpress, is a mature technology with a large number of libraries to extend its functionalities. For maintenance, the engineering team will be responsible and since Django is already used to back the &lt;a href="https://search.creativecommons.org/"&gt;CC Search&lt;/a&gt;, currently the most ambitious products of the organization, then there should be no problem with this point. Given all this it seemed obvious to me to choose Django for this project.&lt;/p&gt;
&lt;h3 id="next-steps"&gt;Next steps&lt;/h3&gt;&lt;p&gt;It has only been a week so we're just starting, so next task to do sooner are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Conduct interview with staff to iterate over designs&lt;/li&gt;
&lt;li&gt;Setup the Django project&lt;/li&gt;
&lt;li&gt;Model the legal database&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="how-you-can-help"&gt;How you can help&lt;/h3&gt;&lt;p&gt;We are still discussing the new image before start coding so these are some options if you want to get involved:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Suggest improvements to designs/UX or new features joining the discussion on the &lt;a href="https://creativecommons.slack.com/channels/cc-dev-legal-database"&gt;&lt;code&gt;#cc-dev-legal-database&lt;/code&gt;&lt;/a&gt; Slack channel or on the &lt;a href="https://github.com/creativecommons/legaldb"&gt;github&lt;/a&gt; project.&lt;/li&gt;
&lt;li&gt;Contribute directly to the CC's Vocabulary design system on &lt;a href="https://github.com/creativecommons/vocabulary"&gt;github&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;Submit a case or a scholarship related to CC licenses on the &lt;a href="https://labs.creativecommons.org/caselaw/contribute/"&gt;current site&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;CC Legal Database&lt;/strong&gt; is my Outreachy project for this summer, under the guidance of Director of Engineering &lt;a href="https://creativecommons.org/author/kriticreativecommons-org/"&gt;Kriti Godey&lt;/a&gt; and Core Systems Manager &lt;a href="https://creativecommons.org/author/timidcreativecommons-org/"&gt;Timid Robot Zehta&lt;/a&gt;, they are very supportive and the CC community is notable welcoming.&lt;/p&gt;
&lt;p&gt;This is the first post of a series so stay tuned if you're interested to know more.&lt;/p&gt;
</content></entry></feed>