GSoC [06]: Making It Through Second Evaluations

My second evaluations for Google Summer of Code completed yesterday, along with that the pipeline for the project has been completed. Let’s not move too fast, and slowly get to completion of Nephos’ second evaluation.

Nephos is being developed in a modular manner; with data piped into different through a common database stored locally in the system. For example, once a recording is done, the important details for the same are saved in the database with a tag “not processed”. Similar tags are used to mark different stages under which the recording is, and different modules fetch only the files marked for them.

The benefit of separating all modules is that testing becomes a lot easier on the local system, as well as it becomes easy to write unittests for the same. Module approach shall be helpful in the future in catching bugs, and rectifying them without ruining or looking at the entire Nephos source code.

The third week of this coding period begin with finishing the work on preprocessing module; it was made as configurable as possible: the user was allowed to set arguments for video conversion as well as subtitle’s extraction. A necessary configurable option to set path to required binary files was also provided in the configuration files.

Soon after the processing module, I started with the uploader module. With the help of my fellow GSoC student developer Aaditya Nair, who has been working on the same project along with me, it was a piece of cake to complete the code of the module.

There was a major issue which prevented the use of multi-threading; authentication and building of drive API was required for each thread separately. Apparently, the built drive service wasn’t being shared between the threads as I had expected it to. The authentication method was refactored to fix the bug; now each thread has a separate service built exclusively for it. I plan to improve this aspect, and find a method so authentication is done only once.

Once the modules were complete, I added them to the main pipeline and testing began; the most tensed tests I’ve ever done. It was yet easier to test on my system, the testing on server was more tight and time-consuming. Many bugs were fixed in this phase and entire code was made compatible to run on Python 3.4, Python 3.5 and Python 3.6.

Before the second evaluations begin, I was ready with a working pipeline (with bugs I realised later). My mentor asked me to do rigorous recordings for next 24-48 hours and fix all the bugs. He was to monitor the progress through the uploads made. Also during the evaluations, processing of recordings was moved to an external shell script to make it more customisable and future proof.

I did pass the second evaluations with good feedback from my mentor after a rigorous two day monitoring and fixing crucial works. Now, Nephos is capable of recording, processing and uploading to google drive. As promised in the proposal, uploading module has been created as an Interface and more online cloud services can be integrated easily.

For the past two weeks I kept myself focused only on completing the pipeline of Nephos, and all the major source code, and hence have been neglecting documentation and writing unittests. They have been shifted to third phase along with the incorporation of multiple new features as put forward by my mentor; in this final phase, he’ll be closely monitoring our works and suggesting changes accordingly, everyday!