Generating a tool chain tutorial for the MSUI user interface by automation operations : GSoC 2021
Student : Hrithik Kumar Verma
Organisation : Python Software Foundation
Sub-Organisation : Mission Support System (MSS)
Mentors : Reimar Bauer, May Bär, Jörn Ungermann, Tanish Grover, Christian Rolf
You can see the project listed on GSoC website here
You can glance through my weekly blogs on python-gsoc website here
You can checkout my full proposal here !
The Mission Support System (or short MSS) is a piece of software that scientists and meteorologists use to plan atmospheric research flights. It is a tool that shows weather conditions and other parameters. The scientists place waypoints for research flights using various sets of forecast data from web map services.
The current documentation for the Mission Support System client software covers the basics of installation and provides a brief overview of how to use the MSS client UI, but it lacks a detailed and comprehensive explanation of all of the User Interface's aspects and functionality. The existing documentation was weak in terms of practical application of the software.
In this project, the basic concept was to make an automated video tutorial. Manual recording of the software, on the other hand, would be a bad idea because the MSS software is constantly being developed and updated. Sometimes new features are added, or the entire graphical user interface changes, and in such cases, the manual video guide would be out of date and useless. To keep up with changes, the video tutorial must be automated so that by updating its scripts, the video tutorial and guide update automatically. It is the same as updating the tests or a feature.
- To automate the use of MSS software according to various use-case scenarios.
- To explain to the user and acquaint with them the various functionalities and features of MSS.
- Record the automation of the software and generate the same in terms of required video codecs.
- Generate text files for demonstrating the motive performed in the video.
- To post-process the automated tutorials with audio and subtitles.
- To render the tutorials for optimized quality and performance with tools like ffmpeg.
- To add to the
Helpsection, the automatically generated videos.
The documentation for the project can be found here. It is basically a user guide of how to proceed with the tutorials.
You can view my created videos and (automatically created with a set of python programs) and it's associated license uploaded on a public share.
After the next major release you will find the tutorials in the help section of the MSS documentation.
-
All of my work has to be made cross platform usable and I had to find the solutions for some QT or platform differences.
-
The pixels of the User Interface in different OSs were different and it created major problems during image recognition in the tutorial programs and hence this has to be handled differently for different OSs.
-
Handling of the multiple windows opened when automation runs because different systems has different type of shortcuts enabled for their system. Tried to optimize them as much as possible.
-
I developed a screen recorder module which generated the present cursor image and recorded with capturing cursor/mouse position making the same visible in the automatically generated tutorials.
-
During my project timeline, the topview UI was changed. And I updated my code in conjuction with the updated UI. By this, I proved the concept of an easily done, updated new tutorial.
-
With the update of new UI coming soon in future releases, I created an automated tutorial script about Mscolab according to my GSoC mate's branch. Since, it's been brought forward in the mainframe UI, this tutorial would be helpful in getting acquainted with the upcoming UI sooner and better. In this too, the project's potential can be represented.
-
Written the technical and explanatory things in text files and made modules to translate them into different languages of choice and converting the texts to speech for available languages in t2s APIs. Also, I have given the user a free space to translate and perform t2s according to the API of his/her choice.
| Tutorial Name | Purpose |
|---|---|
| Waypoints tutorial | To make the user aware of how to add, move, delete waypoints ; Zoom, pan into the map ; saving the flightrack ; switching to previous and next map structure. |
| Web Map Service tutorial | To make the user know how to connect to the web map service server, get the weather layer list capabilities, multilayering of layers, starring of the layers and filtering the layers as starred or with a particular name, how to use auto-update feature, caching feature, how to remove a layer from map, and other specific technicalities associated with a particular layer. |
| Views tutorial | Using topview and sideview together and demonstrating user how changes take place immediately in all views, using both views with their web map service, moving and adding waypoints in sideview, inserting data about places in tableview along with clone, insert, delete and reverse waypoints, showing linear view and it's web map service and traversing through it's different layers and changing the flightrack accordingly. |
| Remote Sensing tutorial | To give user a basic idea of what remote sensing is in flight planning, to tell the user MSS has remote sensing and can calculate solar angles of Sun, Venus, and Moon with respect to azimuth and elevation angles, draw tangents for some operations at a particular distance and of a particular color in estimating various parameters peculiar to flight planning. |
| Satellite Track tutorial | To give the user a hint that positions of satellites from satellite prediction files data can be found in MSS and the various time and dateframes in the prediction files can be selected as per requirement. |
| KML tutorial | how to add .kml files in layerlist of KML overlay, how to make a kml file visible/hidden on the map, how to select/unselect all files, how to change color and linewidth of a particular KML file. |
| Hexagon Control tutorial | how to add hexagon of waypoints around a certain waypoint in table view and see the changes visible in topview, putting center latitude and longitude of the central waypoint and changing the radius of the hexagon, surfing through different values of angle of first point and noticing the subsequent changes in the topview. |
| Performance Settings tutorial | how to add performance settings json file in table view, how to show/hide the performance data, how to change the performance (of aircrafts) data such as takeoff time, maximum takeoff weight, aircraft weight. |
| Mscolab tutorial | how to add a new user, logging in to the mscolab server, how to create a new shared project, how to activate the project, adding users and specifying/changing their access roles as admin, collaborator or viewer, checking out to a particular version history of change and creating named changes, chat within the group and upload items for all to see, working locally if one wants and then saving that change to server or fetching change from server, switching between local flight track and shared flight track, delete the user or log out of server. |
A Note: Don't confuse the tutorial videos as manually created. They were automatically created via python programs and recorded with a screen recorder module. I tried to automate as much as possible, all the things. The automated videos were then trimmed, reduced in size via command line arguments in ffmpeg and the audios were also automatically generated by scripts and can be later merged into the video through ffmpeg command line instructions.
The following is a table for all the Pull Requests that I have requested into the branch GSOC2021-HrithikKumarVerma. All my PRs have been merged into this branch and they are ordered below in ascending order of time. There is also one closed PR.
| Code Contributions | Link to the relevant PR |
|---|---|
| Screen recorder module and started my first tutorial, that is, waypoints tutorial (PR was closed) | https://github.com/Open-MSS/MSS/pull/1040 |
| Completed the waypoints tutorial and fixed screen recording problems | https://github.com/Open-MSS/MSS/pull/1067 |
| Optimized waypoints tutorial for Linux systems | https://github.com/Open-MSS/MSS/pull/1074 |
| Created the web map service tutorials | https://github.com/Open-MSS/MSS/pull/1083 |
| Views tutorial covering topview, sideview, tableview and linearview simultaneously | https://github.com/Open-MSS/MSS/pull/1104 |
| Remote Sensing and satellite track prediction tutorial, added option to record a particular area | https://github.com/Open-MSS/MSS/pull/1136 |
| Completed KML overlay, Hexagon Control and Performance settings tutorial, fixed bugs | https://github.com/Open-MSS/MSS/pull/1144 |
| Mscolab tutorial, text translation and text-to-speech feature for generating audios | https://github.com/Open-MSS/MSS/pull/1162 |
| Updated the documentation with tutorials.rst file and added some text files | https://github.com/Open-MSS/MSS/pull/1166 |
| Added text files for kml, mscolab, wms and views tutorials | https://github.com/Open-MSS/MSS/pull/1170 |
The final outcomes are a toolchain series of tutorials that helps the end user to know about the various functions and features that Mission Support System has to offer in scientifically researched flight planning. These tutorials will be embedded in the Help section of the main UI of MSS in upcoming releases. The tutorials are a great tool for newbies to explore the complex software and make themselves aware about the hidden features of MSS and hence resulting in the increase of their confidence to use MSS more easily, efficeintly and egnormically! The wide varieties of audio languages and subtitles to be offered in the tutorials will help users at an international level in understanding MSS.
The automated tutorials can be viewed online in the Folder "Tool Chain for Automated Tutorials by Hrithik Kumar Verma."
A screenshot of the same directory has been shared below:
Small parts of my proposal have not been completed such as:
- Mixing of audio and video has to be done via CLI based on the descriptions.
- We postponed uploads to youtube to the next major release.
- Tests for the screenrecorder
- The idea of changing the UI for a video tutorials section has been also postponed for the next major release.
I have already mentioned ffmpeg commands in the documentation for merging audios into the tutorial videos. They are one to two lines command line arguments.
Almost all objectives have been completed in this project but there are a few things as can be seen from a developer's eye that can be improved and amalgamated in the present tool chain of tutorials. As the saying goes, there is always a scope for improvement and the most complex problems can be solved through computer science but the only deciding criteria is enough time. Since, tool chain for tutorials is a very new feature, there were somethings that has been left out and many things to improvise in the future. Listing some as:
- The screen recorder window which gets closed only when it is in focus. This thing can be made more simpler for the developer and also for the program if it gets closed in the background when we want so.
- When the tutorials are being automated on the system, the display is busy at the time and the user cannot perform any other operation. For this, some ideas have been conceived during the GSoC period of using PyVirtualDisplay and this can be improved by making the tutorials run in a virtual display.
- There are a varierty of text-to-speech and text translation modules which are better in terms of phonetics and pronounciations, and hence can be implemented according to the developer's choice.
- There can be certain functions in the utils module of MSS to return the images and other OS related stuffs for making the code more optimized and efficient.
- The uploading of the videos automatically on Youtube via it's API and including more languages to cater to wide linguistic groups and people all around the world in a more easier way than now.
The Google Summer of Code 2021 was a thrilling and amazing experience for me. As I write this now, I think I was a different person three months ago and I am a different person now. I got so much to learn, love, experience, and enjoy as a python software developer. The confidence and skills increased inside me manyfolds. It's hard for me to believe that I have developed a feature in a software that so many people are going to use. Really exciting!
For this, I am really thankful to all my mentors - Reimar Bauer, May Bär, Jörn Ungermann, Tanish Grover, and Christian Rolf. I would appreciate and admire them for patiently reviewing my code, giving me constructive suggestions, and guiding a novice like me throughout the program. They were always there for me whenever I came across any blockers or got stuck somewhere. Without their help and support, this programme wouldn't have been a success!
I learnt many different things about technologies, best coding practices, and how to plan, organize, communicate and think. It was a wonderful experience working with Python Software Foundation and building things for Mission Support System. All said and done, I want to thank Google for providing me and many others this wonderful oppurtunity. This has strengthened my affinity towards Open Source much more and even post GSoC, I would always be an indelible part of the community and MSS.