Merge pull request creativecommons#380 from creativecommons/update_master

Update master

Author: Dhruvi16

.gitignore

@@ -13,3 +13,4 @@ node_modules
 gui/build
 gui/node_modules
 gui/static/gen
+*.log

content/blog/entries/linked-commons-autocomplete-feature/autocomplete-feat-in-action.gif

@@ -0,0 +1,83 @@
+title: Linked Commons: Autocomplete Feature
+---
+categories:
+announcements
+cc-catalog
+product
+gsoc
+gsoc-2020
+open-source
+cc-dataviz
+---
+author: subhamX
+---
+series: gsoc-2020-dataviz
+---
+pub_date: 2020-07-31
+---
+
+body:
+## Introduction
+The following blog intends to explain the very recent feature integrated to the Linked Commons. Be it the giant Google Search or any small website having a form field, everyone wishes to predict what’s on the user’s mind. For every keystroke, a nice search bar always renders some possible options the user could be looking for. The core ideology behind having this feature is — *do as much work as possible for the user!*
+
+
+
+<div style="text-align: center; width: 100%;">
+    <figure>
+        <img src="autocomplete-feat-in-action.gif" alt="autocomplete-feature" style="border: 1px solid black">
+        <figcaption style="font-weight: 500;">Autocomplete feature in action</figcaption>
+    </figure>
+</div>
+
+## Motivation
+One of the newest features integrated last month into Linked Commons is Filtering by node name. Here a user can search for his/her favourite node and explore all its neighbours. Since the list is very big, it was self-evident for us to have a text box (and not a drop-down) where the user is supposed to type the node name.
+
+Some of the reasons why to have a text box or filtering by node option.
+- Some of the node names are very uncommon and lengthy. There is a high probability of misspelling it.
+- Submitting the form and getting a response of “Node doesn’t exist” isn’t a very good user flow, and we want to minimise such incidents.
+
+Also, on a side note giving a search bar to the user and giving no hints is ruthless. We all need recommendations and guess what linked commons got you covered! Now for every keystroke, we load a bunch of node names which you might be looking for. ;) 
+
+
+## Problem
+The autocomplete feature on a very basic level aims to predict the rest of a word the user is typing. A possible implementation is though the linear traversal of all the nodes in the list. It will be having a **linear time complexity**. It’s not very good and it’s very obvious to look for a faster and more efficient way. Also, even if for once we neglect the **time complexity**, looking for the best 10 nodes out of these millions on the client's machine is not at all a good idea; it will cause throttling and will result in performance drops. 
+On the other hand, a **trie based solution** is more efficient for sure but still, we cannot do this indexing on the client machine for the same reasons stated above.
+So far, it is now apparent that we implement this feature on the server and also aim for at least something better than linear time complexity.
+
+
+
+## A non-conventional solution
+We could have used Elastic Search, which is very powerful and has a ton of functionalities but since our needs are very small we wanted to look for other simple alternatives. Moreover, we didn't want to complicate our current architecture by adding an additional framework and libraries.
+
+Taking the above points into consideration we went ahead with the following solution. We store all nodes data into an SQL dB and search for all the nodes whose domain name pattern was matching to the query string. After slicing the query set and other randomization we sent the payload to the client. To make it more robust, we are caching the results in the frontend to avoid multiple calls for the same query. It will surely reduce the load from the server and also give a faster response.
+
+
+## Results
+To make sure our solution works well, we performed load tests, checking that any response time does not exceed 1000 ms. We used locust which is a user load testing tool. We simulated with **1000 users** and **10 as Hatch rate**. 
+The following test is performed on the local machine to ensure that the server location isn’t affecting the results. 
+
+Here are some aggregated result statistics.
+
+
+
+| Field Name              	|  Value        |
+|---------------------------|---------------|
+| Request Count 	        |** 23323     **|
+| Failure Count 	        |**  0 	      **|
+| Median Response Time      |** 360 ms    **|
+| Average Response Time     |** 586.289 ms**|
+| Min Response Time         |** 4.03094 ms**|
+| Max Response Time         |** 4216 ms   **|
+| Average Content Size      |** 528.667 ms**|
+| Requests/s                |** 171.754   **|
+| Max Requests/s            |** 214       **|
+| Failures/s              	|** 0         **|
+
+Since SQLlite has a serverless design, disk io usually has a significant impact on the performance. The above results were executed on a server with HDD storage. Linked Commons server is equipped with faster disk io. It will certainly improve the performance but will be countered by the network latency and other factors like the number of nodes in the dB. So the above results to some degree resemble the actual performance.
+
+
+## Next steps
+In the next blog, we will be covering the long awaited data update and the new architecture.
+
+## Conclusion
+Overall, I enjoyed working on this feature and it was a great learning experience. This feature has been successfully integrated to the development version, do check it out. Now that you have read this blog till the end, I hope that you enjoyed it. For more information please visit our [Github repo](https://github.com/creativecommons/cccatalog-dataviz/). We are looking forward to hearing from you on linked commons. Our [slack](https://creativecommons.slack.com/channels/cc-dev-cc-catalog-viz) doors are always open to you, see you there. :)

content/blog/entries/linked-commons-autocomplete-feature/contents.lr

@@ -0,0 +1,5 @@
+_model: redirect
+---
+target: https://cc-vocabulary.netlify.app/
+---
+_discoverable: no

content/vocabulary/contents.lr

@@ -26,7 +26,7 @@
                 },
                 {
                     "name": "Aksh Gupta",
-                    "role": "Project Member"
+                    "role": "Project Collaborator"
                 },
                 {
                     "name": "Shubham Pandey",
@@ -66,7 +66,7 @@
                 },
                 {
                     "name": "Abhishek Naidu",
-                    "role": "Project Member"
+                    "role": "Project Collaborator"
                 }
             ],
             "name": "CC Search",
@@ -204,15 +204,15 @@
                 },
                 {
                     "name": "Chidiebere Onyegbuchulem",
-                    "role": "Project Member"
+                    "role": "Project Collaborator"
                 },
                 {
                     "name": "Ayan Choudhary",
-                    "role": "Project Member"
+                    "role": "Project Collaborator"
                 },
                 {
                     "name": "Tanuj Agarwal",
-                    "role": "Project Member"
+                    "role": "Project Collaborator"
                 },
                 {
                     "name": "Jahnvi Gupta",

databags/community_team_members.json

@@ -14,44 +14,34 @@
                     "name": "Move data cleaning pipeline from API to Catalog"
                 },
                 {
-                    "description": "Manage Catalog deployment and provisioning entirely through infrastructure as code.",
-                    "gid": "1167425798148811",
-                    "name": "Improve Catalog Deployment and Provisioning"
-                },
-                {
-                    "description": "Create better documentation for community contributors by consolidating internal and public documentation and making it available for everyone.",
-                    "gid": "1167425798148815",
-                    "name": "Improve Documentation for Community Contributors"
+                    "description": "Update Catalog schema to include new metadata generated through AWS Rekognition.",
+                    "gid": "1154270978154717",
+                    "name": "Implement architecture for schema for new metadata [AWS Grant]"
                 },
                 {
                     "description": "Plan out search algorithm changes to incorporate image metadata generated via AWS Rekognition.",
                     "gid": "1154270978154720",
                     "name": "Plan search algorithm changes for new metadata [AWS Grant]"
                 },
-                {
-                    "description": "Update Catalog schema to include new metadata generated through AWS Rekognition.",
-                    "gid": "1154270978154717",
-                    "name": "Implement architecture for schema for new metadata [AWS Grant]"
-                },
                 {
                     "description": "Improve how and where we explain licenses, and consider ways to make it easier for reusers to understand and comply with license requirements.",
                     "gid": "1147666754358269",
                     "name": "License Explanation/Compliance Improvements"
                 },
                 {
-                    "description": "Improve the support pages on CC Search, which includes the Collections page, for a better experience. Add explanation text for collections, improve flow.",
-                    "gid": "1149385618454685",
-                    "name": "Improved Support Pages"
+                    "description": "Offline Old Search (oldsearch.creativecommons.org) and redirect traffic to CC Search. Prior to this, build in messaging on Old Search, and support similar functionality on CC Search. See \"Meta Search Integration\" for related work.",
+                    "gid": "1149456632174214",
+                    "name": "Offline old CC Search"
                 },
                 {
-                    "description": "Integrating meta search functionality into CC Search for sources that are not currently indexed, and content types we do not currently support.",
-                    "gid": "1174575887784290",
-                    "name": "Design Sprint: Meta Search Integration"
+                    "description": "Research and test potential integrations for Web Monetization into CC Search and other CC web properties.",
+                    "gid": "1153114910798067",
+                    "name": "Web Monetization: Phase 1"
                 },
                 {
-                    "description": "Offline Old Search (oldsearch.creativecommons.org) and redirect traffic to CC Search. Prior to this, build in messaging on Old Search, and support similar functionality on CC Search. See \"Meta Search Integration\" for related work.",
-                    "gid": "1149456632174214",
-                    "name": "Offline old CC Search"
+                    "description": "Improve the support pages on CC Search, which includes the Collections page, for a better experience. Add explanation text for collections, improve flow.",
+                    "gid": "1149385618454685",
+                    "name": "Improved Support Pages"
                 },
                 {
                     "description": "Make accessibility improvements to the UI.",
@@ -63,6 +53,11 @@
                     "gid": "1149456632174198",
                     "name": "Internationalization Infrastructure"
                 },
+                {
+                    "description": "Update our Common Crawl provider infrastructure to:\n(1) use Apache Airflow instead of AWS tools like Data Pipeline & Glue for processing data\n(2) unify provider processing to use the same base classes as API providers",
+                    "gid": "1167425798148813",
+                    "name": "Improve Common Crawl Infrastructure"
+                },
                 {
                     "description": "Designing and prototyping an upcoming user interface for searching for audio on CC Search.",
                     "gid": "1163392248010945",
@@ -73,86 +68,66 @@
                     "gid": "1171015130050099",
                     "name": "Audio Support and Integration"
                 },
-                {
-                    "description": "Update our Common Crawl provider infrastructure to:\n(1) use Apache Airflow instead of AWS tools like Data Pipeline & Glue for processing data\n(2) unify provider processing to use the same base classes as API providers",
-                    "gid": "1167425798148813",
-                    "name": "Improve Common Crawl Infrastructure"
-                },
-                {
-                    "description": "Switch our Catalog data ingestion for Wikimedia Commons to use the data dumps provided by Wikimedia instead of the MediaWiki API.",
-                    "gid": "1167425798148807",
-                    "name": "Use Data Dumps for Wikimedia Ingestion"
-                },
-                {
-                    "description": "Research and test potential integrations for Web Monetization into CC Search and other CC web properties.",
-                    "gid": "1153114910798067",
-                    "name": "Web Monetization: Phase 1"
-                },
-                {
-                    "description": "Build a UI for the Catalog API, where users can sign up, manage access, see usage metrics and statistics.",
-                    "gid": "1149478266493761",
-                    "name": "API UI with Usage Dashboard"
-                },
-                {
-                    "description": "Make CC Catalog API documentation more accessible to CC Search users, and improve user experience.",
-                    "gid": "1164969092703369",
-                    "name": "API documentation improvements"
-                },
                 {
                     "description": "Store a private copy of all the images in the CC Catalog to analyze via machine learning.",
                     "gid": "1154270978154722",
-                    "name": "Scraping & Resizing Work [AWS Grant]"
+                    "name": "Scraping & Resizing Work for Rekognition [AWS Grant]"
                 },
                 {
-                    "description": "Collect and use structured data from Wikidata to enhance our search algorithm with semantic search.",
-                    "gid": "1167425798148823",
-                    "name": "Wikidata integration with Catalog & Search Algorithm"
-                },
-                {
-                    "description": "Build an analytics UI that is fed by Google Analytics and our internal analytics database.",
-                    "gid": "1149385618454692",
-                    "name": "Usage/Reuse Metrics Dashboard"
+                    "description": "Generate metadata via machine learning (using AWS Rekognition) on a set of ~100 million high quality images from the CC Catalog.",
+                    "gid": "1154270978154727",
+                    "name": "Run Rekognition on 100m images [AWS Grant]"
                 },
                 {
                     "description": "For all possible providers, use their APIs to ingest data into the CC Catalog instead of scraping websites via Common Crawl data.",
                     "gid": "1149385618454708",
                     "name": "Switch from Common Crawl to API"
-                },
+                }
+            ]
+        },
+        {
+            "name": "Q4 2020",
+            "tasks": [
                 {
-                    "description": "Generate metadata via machine learning (using AWS Rekognition) on a set of ~100 million high quality images from the CC Catalog.",
-                    "gid": "1154270978154727",
-                    "name": "Run Rekognition on 100m images [AWS Grant]"
+                    "description": null,
+                    "gid": "1186693612765822",
+                    "name": "Search Relevance Improvements: Language Analysis, Quality Metrics, Minimums"
                 },
                 {
-                    "description": "Upgrade the CC Catalog database to use a schema-less database instead of the relational database (Postgres) that we currently use.",
-                    "gid": "1167425798148817",
-                    "name": "Upgrade Catalog:  Data Lake"
+                    "description": "Design updates to the CC Search UI in response to new metadata available as a result of applying machine learning to selected images in the Catalog. At a minimum, we expect new filters will be an option. Integration of design will take place subsequently.",
+                    "gid": "1154270978154729",
+                    "name": "Plan UI Updates in Response to Metadata [AWS Grant]"
                 },
                 {
                     "description": "Automate the process of finding new providers of CC-licensed content to index into the CC Catalog.",
                     "gid": "1167425798148819",
                     "name": "Provider Review Automation"
                 },
                 {
-                    "description": "Implement changes to CC Search (frontend) and Catalog to make use of thumbnails, as they become available.",
-                    "gid": "1154270978154725",
-                    "name": "Implement Use of Thumbnails in Search & Catalog [AWS Grant]"
+                    "description": "Build an analytics UI that is fed by Google Analytics and our internal analytics database.",
+                    "gid": "1149385618454692",
+                    "name": "Usage/Reuse Metrics Dashboard"
                 },
                 {
-                    "description": "Prepare partnership guidelines for CC Search. Create a page on CC Search publishing these guidelines.",
-                    "gid": "1146971105237802",
-                    "name": "Partnership guidelines for all integration types"
+                    "description": "Once the Rekognition crawl finishes, we want to crawl the rest of the catalog (but not feed them to rekognition). This will give us useful metadata like dimensions and quality.",
+                    "gid": "1186693612765814",
+                    "name": "Scrape all images and set up feed for new ones"
                 },
                 {
-                    "description": "Design updates to the CC Search UI in response to new metadata available as a result of applying machine learning to selected images in the Catalog. At a minimum, we expect new filters will be an option. Integration of design will take place subsequently.",
-                    "gid": "1154270978154729",
-                    "name": "Plan UI Updates in Response to Metadata [AWS Grant]"
-                }
-            ]
-        },
-        {
-            "name": "Q4 2020",
-            "tasks": [
+                    "description": "Create better documentation for community contributors by consolidating internal and public documentation and making it available for everyone.",
+                    "gid": "1167425798148815",
+                    "name": "Improve Documentation for Community Contributors"
+                },
+                {
+                    "description": "Manage Catalog deployment and provisioning entirely through infrastructure as code.",
+                    "gid": "1167425798148811",
+                    "name": "Improve Catalog Deployment and Provisioning"
+                },
+                {
+                    "description": "Make CC Catalog API documentation more accessible to CC Search users, and improve user experience.",
+                    "gid": "1164969092703369",
+                    "name": "API documentation improvements"
+                },
                 {
                     "description": "Design and build an embed of CC Search that can be placed on any website, as a starting point to discover objects in CC Search. Components from Design Library must be used, with the goal of simplicity.",
                     "gid": "1168725971351188",