GSoC Devlog

Third Phase

14 July

IT’S PHASE III

13 July 2018 - 30 July 2018 So, Yes, I did pass the Phase-II. This is when I should thank my mentor (@gaacegege) and the community. Thank You!. The last phase was all about getting the client up and running and publishing it to the VSCode Marketplace. We did it. Now, VSCode can fix all your issues right from your editor window and you can almost one click install the client from the store. This phase is all about optimizing coala and coala-vs-code. I now have a almost agreed upon cEP! Anyway here are some developments from the past two weeks.

cEP
coala uses something called cEP as a proposal mechanism to propose changes to the coala core. This is so the community can know about and vote on major changes. Typically cEPs are required during feature updates and architectural changes. Optimizing coala for coala-ls requires at least one feature extension and one architectural change. I intend to introduce a feature known as tagging, so you can conditionally execute a collection of sections based on the invoking tags. The aforementioned change to the architecture has to do with supporting coala execution on in-memory versions of a file rather than their disk counterparts.
This cEP has been stuck in review for a month now, I have requested for initial approval so I can start working on the implementation, thanks to @shreyans800755 I have approval on implementation model of the tagging feature upgrade and thanks to @jayvdb I have initial approval on the cEP and can now start working on the implementation. Follow the cEP developments here
Prototypes
Improving the cEP involved writing at least prototype implementations of the proposed changes. I have thus implemented a prototype of the proposed tagging feature and it functions well enough, prototype implementation of FileProxy support was also found to be functional enough.
PyDelhi and coala Although not strictly related to this blog, I did represent coala at this months meetup of PyDelhi, I helped some newcomers get onboard, some Atleast ~10 newcomer PR’s were submitted and some even got merged.

In all, the past two weeks have been mostly about working on the cEP and getting it into an acceptable shape so I can start implementing the proposal.

Second Phase

5 July

The End of Times.

04 July 2018 - 10 July 2018

So, the end of Phase-2 of GSoC 18 is about to begin in a few hours, It feels like the second phase passed blazing fast. This period got more packed than I could have imagined mostly due to some personal things that required my attention. I feel it quite important to mention that my mentor @gaocegege was quite positive about it and helped me in every manner possible even when his schedule got a bit too packed. @gaocegege Thanks!. Second phase was mostly about supporting formatting and producing a vscode client for coalals. I think we achieved those goals to an acceptable level, although we have our PR’s pending for review, the architecture was discussed beforehand so no major issues must’ve propagated through. Currently, the newer version of coala-vs-code is installable from the vscode marketplace you will be required to install coalals manually and merge the code on you local dev env to test it out. This should flawlessly work when coalals is made available via PyPi.

Developments

Here are the developments from the last week of development specifically.

Support window/showMessageRequest: window/showMessageRequest can be used to alert the end user about some event or an option and the feedback can be collected and processed. The support comes to the LangServer class but is currently not used in any place as is. This could be useful for future developments such as in event of Phase-3.
Tests coala-ls yet again has complete coverage of tests. It now features almost 80+ tests.
Docs coala-ls also features complete docs.
Performance Metrics We have the performance metrics in place for the additional features introduced in this phase i.e textDocument/formatting.
coala-vs-code tests Because of the pending PR status of coalals, End to End tests are tough to design because of the local merges required. Hence, basic unit tests are available for coala-vs-code.
Demonstration Video As described in the requirements, we have a video demonstrating the capabilities of the coala-ls and coala-vs-code.

Links

coala-ls: http://github.com/coala/coala-ls/pulls/
coala-vs-code: https://github.com/coala/coala-vs-code/pulls
Videos: https://www.youtube.com/watch?v=MeybdlCB96U & https://www.youtube.com/watch?v=uOnFQqe0IjU

Huh, we are here, this marks the end of the blog post. Wish we luck for the evals.

28 June

Code, Write.

28 June 2018 - 4 July 2018

This is a backdated post. This is going to be fun. So, last week I had textDocument/formatting working so you can now fix some/all of your code linting issues. This week it was the time to begin development of the vscode plugin for the coala-ls, so you, yes YOU, can use coala more efficiently via the coala-ls I was developing. The good news is we do not need many changes to the existing vs-code client, the one @gaocegege developed back in the day. The worst part is vscode plugin needs to be written in TypeScript. I have not been a big user or proponent of node, so I never really learnt this famous type checked flavor of JavaScript. Well, its apparent now I think, I will have to learn at least some TS. So, here we go, this is a brief of all the developments happened during the week.

Developments

textDocument/formatting: Completed the support for formatting, coala-ls now has docs, tests and complete support for formatting.
coala-vs-code cleanup: Earlier the repository used to contain a language server built specifically for vscode, from now on it will be used to host the language server client. Repository wide clean up is thus required.
coala-vs-code extension Improves the extension by allowing to be user/workspace configurable, currently supports coalals.maxWorkers and coalals.tcpPort. It also is invoked for all the supported languages. This filter will further finished during phase 3 when some cEP proposals are implemented.
Completed cEP Finally, the long standing cEP is now complete and has been put up for review. The cEP is about optimizing coala core for being better compatible with language server. The current proposal is so designed to produce minimal changes to the coala core. It is now cEP-0028.

To summarize, I worked with node, TS, completed my cEP and got the vscode client working in both TCP and IO modes.

27 June

Second Phase, Yay!.

12 July 2018 - 27 July 2018

So, I passed the first phase. I would like to take a moment to thank my mentors especially @gaocegege and the org admin @jayvdb. All the mentors and fellow students who helped me build my language server, thank you! I am gald that I have passed phase one. It is now time to improve the language server to support yet some more request types on the server and polish the VS Code plugin. This blog post accounts for the past two weeks. Now that I am finally done with my exams and the vacations have started I travelled back to my home and started working on the tasks for Phase 2. As mentioned above Phase 2 involves improving the server and working on the client. I picked working on expanding the server request type support to begin with. Our prime goal was to support textDoument/formatting and a few others. formatting is intended to fix the issues related to spacing and indentation in the code but initially coala ls will fixing all the code irrespective of the type of the patch. This can get tricky at times when the diffs are overlapping. I am confident that some additional changes introduced during Phase 3 will solve those issues and make formatting a feature to be used regularly.

Developments

The past two weeks mainly involved researching about the formatting and completing some back logs. Anyway, here are a few developments.

Refactoring: The coalals.results module was refactored to make it make into a sub module, now with coalals.results.diagnostics.
Support code fixes: Added coalals.results.fixes, now the diffs generated by the coala core can be parsed into coalaPatch objects.
Misc Changes: Some additional changes have been introduced wherever required to support the new formatting request type.
Supports textDocument/formatting: coala language server now supports formatting requests. Initially this does not fix all the issues but skips any overlapping changes. Here is a sneak peak: ()[https://youtu.be/uOnFQqe0IjU]
Adds Tests: The coverage of the tests has been bumped to 100%. All the newly added code is covered by the tests too.
Docs: As usual, all the code has been properly documented.

We now have a working implementation of code fixing in the editor itself using coala and the formatting requests. My mentor and I have decided on how to implement and work with the VS Code plugin.

First Phase

11 June

Fourth Week Ends

4 June 2018 - 11 June 2018

This marks the end of week 4, and by extension also means that now the evaluations for phase one are open. We had a lot of meetings this week to make sure that we are covered for the evals. I had a few minor tasks that I totally forgot about but @gaocegege made sure I completed them. One thing I feel obliged to write about is how I screwed up again by pushing a huge tangled mess of code without any neat structure. Again, my mentor was cool enough about it and dragged me through the situation and helped me restructure the code to an acceptable form. I think I will make it a priority to not repeat any such behavior and keep my code changes atomic enough.

Developments

This week included a lot of code refactoring and all the other stuff you do to it to make it more readable and maintainable.

Commit Restructuring: As mentioned above I had previously maintained a ‘not-so-logical’ commit history and this was once again turning into a problem for getting the code rapidly reviewed and merged. I fixed all of the commit history to at least help the situation a tad more.
Performance Metrics: One of our tasks for Phase-1 was to have initial performance metrics for the coala language server. I chose to write a script that automates the raw metric data collection. The data although needs some processing to get it into an insightful form.
m_exit: The language server was missing support for exit notification which was later implemented in this phase.
Tests: The tests were using nix based file paths which could’ve been a problem for testing on Windows, hence they were improved to support an OS independent test suite. Some changes introduced during the code restructuring and supporting the missing notification required extending the tests and improving upon them too.
Docs: One of the important things that I did not touch upon in the early stages of developement was documenting the code. This was a Phase-1 task and I thus added documentation to every single public entity of the project.

I think we are covered for the Phase-1 wvals, even though I had been behind on my schedule more than a few times we pulled off and completed all the tasks on time. Since this could as easily be my last post from GSoC-18, I want to highlight that this has been a great experience If not anything I have improved vastly upon my code management skills which I now realize I was terrible at. I have also got to know a lot of cool people because of this.

Update: I got an email from Google saying I have passed Phase-1! I look forward to work even more tirelessly. Thanks to all for helping me get through.

29 May

Two Weeks Flyby

21 May 2018 - 4 June 2018

A busy two weeks. I had quite a few exams in the first week and the coding period was to begin in the very same week. And because of that I could not put in all the time I wish I had. My mentor was understanding of the fact and told me he was fine with it if I could get back on track by putting more time in the subsequent days. So, I did. Once the first week ended I had a lot time for the next exam. I have three major tasks for this phase, to design the architecture of ls, build a minimal language server and have complete test coverage and docs for it. Because of support from one of my mentors (@gatesn) and the arch we decided upon, the second task’s goals were either almost readily done or were an intersecting part of the first task. So, I did complete writing a minimal functional server by the time of writing this blog along with tests. The codebase is a major rewrite, so I am awaiting review for it as of now. And, as it turns out the jekyll blog template I was using wasn’t as per the guidelines of the org and thus I had to port the entire blog to medium. I intend to keep this version alive until the end of GSoC, but you can follow the medium blog at @ksdme, coala and gsoc.

Development Updates

This was an active week and I had many changes done to the code base as a part of it. The PR’s are awaiting review. Here are the few development updates.

Checkout the PR and codebase at:

https://github.com/coala/coala-ls/pull/2
https://github.com/ksdme/coala-ls/tree/gsoc-phase-1-tests
Project Refactor: The very first thing I did to the codebase was to refactor it heavily. The current implementation was not designed to be extensible. This was fixed by improving the code structure making it more modular and removing any code specific to vs-code.
Added Concurrency: The server now handles the coala analysis requests concurrently. This now makes it possible to work on multiple files at once and not miss any updates during the analysis cycles. The concurrency infrastructure has been heavily abstracted to keep the scheduling of those jobs intelligent enough and keep the future updates to the interface module in phase 3 in mind.
Added FileProxy: During certain notifications such as didChange the changes made to the source files are not yet flushed to the disk and using current coala interface configuration it is not possible to analyse those changes. FileProxies provide a mechanism to support such situations by directly providing access to the file contents and some meta information. FileProxies have been implemented and integrated into the module but it’ll have to wait until the coala inteface improves in Phase 3 before making full use of them.
Partial support for didChange, didClose: The goal for first phase was to extend support for didOpen and didSave. They are currently supported and I have even extended partial support to didChange (updating corresponding proxies) and handling proxies on closing a file using didClose notification.
Tests: Previously, the tests were fragmented as behave tests and unittests. I have rewritten some of those tests and extended them to the code that was now introduced. The test suite is now unified and is written in pytest.

Although I am glad that I have pulled off major goals of the Phase 1, some other important goals are in back logs. I intened to complete most of them in the days remaining before the end of Phase 1. Some of them are:

Docs: The codebase only has some comments thrown around sparingly. I intend to write the docs for the entire code base.
Configure CI: The CI needs to be configured to run the tests etc. I am awaiting for the initial PR to be merged to get working on the second issue.
Design Doc: The official design doc for the architecture is still in WIP and needs to be submitted asap. This has the highest priority in my list as of now.

So, overall I am glad that the server works satisfactorily. I still can’t be sure of it until it passes the review though. But, I am hoping for the best. See you next week!.

22 May

First Week Ends

14 May 2018 - 21 May 2018

Time flies by so fast! It has been one whole week since the coding period has begun. I have been a doing a whole lot of reading up and understanding the code. As per our agreed upon schedule this week was devoted to designing an architecture for the language server and figure out all the tools that we should be using.

Development Updates

So, as I said, this week was all about get to know about the stuff I should know to work on the project efficiently. As part of it I had to come up with a project architecture, gather all the resources necessary. Here are some of the things we achieved this week.

Project Architecture: We have the project architecture designed now which we believe is extensible enough to support our future plans.
Project Repo: We now have a temporary repo to hold our project hosted on my github profile (coala-langserver). Once it matures we will merge it upstream.
Resources: I have now figured out all the tools we will be using. Some of them are listed in the next section.
More discussion about project: We have had more discussion about the project on the PR page of the update to projects.coala.io. My other mentor @gatesn also jumped in and gave some valuable review comments.

Things I learnt this week

Coming up with a project architecture is not as easy as it sounds. You need to have a clear understanding of the goals, usability and the future scope of the project beforehand. Fortunately, my project was based on an existing project and I had some source of inspiration from the python-language server.
Here are some things I learnt along the way this week:

A whole lot about CI: CI is awesome! To decide upon the infrastructure of our project I had to read how CI systems and how to configure some them. Fyi, we will be using Travis and AppVeyor.
Plugin ecosystem design: I was evaluating our need for a plugin based ecosystem for the server. Although the idea sounds like an overkill for this use case but I wanted to leave no stone unturned. As it turns out we won’t be using any. Try pluggy.
Python concurrent module: An awesome concurrency library that abstracts away all the nitty gritty details of parallel execution. (yes, parallel, I know it can do thread and process pools).
LSP: I had to read the specification again to make my understanding of project scope more clear and detailed.

The next week will be devoted to build a basic language server with primitive support for at least some request types. You can follow the project at the repo mentioned above.

Bonding Period

14 May

Bonding Period Ends

26 April 2018 - 13 May 2018

Now that the bonding period has finally come to an end, I thought I should let you know about the developments during the period and my plan ahead. Bonding Period is the time meant to help the students get started with the community, coding practices and discuss the details of the project one plans to follow with the community and their mentors. I had a great time knowing the community and the interesting stuff they do.

Developments

Resolved pending PR’s: I joined the coala community in late March and made a few contributions to projects.coala.io and coala-vs-code of which some were left pending in review. I got all of them resolved and merged.

Finish newcomer process: Since the deadline was too close when I joined the community I was instrcuted to work on my proposal first and worry with the newcomer process later on. I have since completed the newcomer process.
Schedule: I worked with my primary @gaocegege and we have decided on a week on week schedule that I should follow to stay on track.
Work flow: We have discussed about the work flow I need to maintain and things such as tooling etc.
Project description update: The project description page on projects.coala.io has been updated with the new milestones and the feature extensions that have been decided upon during this period.

The bonding period has helped me gain deeper understanding of the project and the expected results. I am now eager to code my way through.

26 April Hey!, This is the first post in the series of logs that I'll be writing documenting the progress of my GSoC 2018 project for coala.