Google Summer of Code 2017 Work Product Submission [RajuKoushik][AboutCode]

Google Summer of Code 2017 Work Product Submission

Student: Raju Koushik Gorantla (@RajuKoushik)

Organisation: AboutCode

Project: Building ScanCode live scan server

---

The aim of the project is to build a project which uses ScanCode as a library in a web and REST API application that allows you to scan code on demand by entering a URL and then store the scan results. We have created a django server with REST API support to ScanCode given direct link to code or repository url (GitHub, Git) or any url containing archive files(zip, tar, tar.gz).

As a part of this we have dealt in building the UrlScan API and LocalUpload API.

• The URLScan API allows the user to enter in a direct link to code or repository url (GitHub, Git) or any url containing archive files(zip, tar, tar.gz). The API segregates the URL using a regex matcher and then clones or downloads the codebase at certain path using an asynchronous background task implemented using redis and celery. The downloaded codebase is now scanned using the scancode library and the resultant scan results are fed into the models of the scancode server.

• The LocalUpload API allows the user to upload files of the codebase. The downloaded codebase is now scanned using the scancode library and the resultant scan results are fed into the models of the scancode server similar to the URLScan API.

Work

The ScanCode server was written in Python using the Django framework.

Initially we had to work on building the models which was an integral part of the whole project. Ensuring normalised models to set up the foundation of the scancode server was the key. So we had to spend a decent amount of time in constructing the models and have improved them from time to time. After this we have initialised the server with basic features. Then we have setup the DRF(Django Rest Framework) which was going to be useful in building the APIs.

Then we have created a POST API endpoint to scan code given direct link to code file OR code zip. Once url of code file is received by the server, it will download it and generate ScanCode report of it in the specified return type format (query param). Once the report is generated, it would be returned back as the API call response.

We have then worked on enhancing the server and implementing storage. Now that we have managed to setup a basic server which is up and running, we have added some important features to the server. We have added the feature to scan code given github or git url. Also we have used a background async queue with the backend as ScanCode processing can take some time considerable amount of time with the increase in the size of the codebase. Finally, we have implemented a way to feed scan results to the Database.

Along with this we have integrated the login and register modules for the users to sign up and login and have managed to build a decent UI for the scancode web app.

Finally we have worked on the Documentation with the help of Sphinx.

Code

scancode server

Project Subpart:

• Adding UrlScanAPI.
• Adding the registration and login module.
• UI of the web app.
• Integrated Documentation using Sphinx.

Code Repository: https://github.com/nexB/scancode-server

Contributions: https://github.com/nexB/scancode-server/commits?author=RajuKoushik

---

The constant support of our mentor Philippe Ombredanne and the contributions of my fellow GSoC partner Ranvir Singh has made it possible to build the scancode server.

Thanks :)