Your first homework assignment is to create a DevOps application workflow for automatically building and analyzing software applications. As part of the DevOps application workflow, you will use the Application Programming Interface (API) of the tool called Understand (https://scitools.com/non-commercial-license/), a static code analysis tool that supports many programming languages and it is used by many Fortune 500 companies. You should apply for a non-commercial license immediately, install the tool, and investigate its IDE and its API libraries. You can complete this homework using a language of your choice, e.g., Java or Scala or Python or Go or Clojure or simply the utility curl when applicable (I prefer that you use Java for this assignment). You will use SBT or Gradle - your choice - for building the project. You can use the latest community version of IntelliJ IDE for this assignment.
The goal of this homework is to gain hands-on experience with DevOps workflows and analyzing the source code of object-oriented applications statically as part of the workflow to obtain information about relationships among variables and types in these applications. You will create a simulation of the DevOps workflow at an enterprise level. Within your simulated enterprise DevOps you will install and configure the GitLab server that is used by many companies to implement a scalable Git-based fully integrated platform for software development (https://docs.gitlab.com/ee/README.html). Then, you will install Jenkins and you will set up Jenkins+Gitlab integration, install Jenkins Jacoco (or Cobertura) plugin for Java code coverage, install Jenkins Job DSL Plugin for programmatically creating Jenkins jobs, and this Jenkins Job DSL Plugin exposes a Jacoco (or Cobertura) interface that you will use to run tests and obtain information about test coverage. The idea is to allow software engineers to create and update software projects by pushing them into your Gitlab server and when doing so to test and analyze the code in the repo automatically and report results.
At a high level, you will simulate the work of software engineers (SE) at an enterprise by pulling commits of Java (or some other language) git projects from Github using its API calls and inserting them into remote repos on your locally installed Gitlab server. The input to your SE simulating application is the language which will be used to filter open-source projects from Github, which is a publically available and it limits the rate of communications from its non-paying clients to 1,000 projects a day. Some API calls, e.g., GET content, have the upper limits of 1,000 files for a directory and 1MB in size. In many cases, the limits that Github imposes on some API calls can be bypassed by using some other API calls, e.g., GET tree recursively. In your DevOps, one component simulates SE by obtaining the content of various Java open-source projects and their attributes, pushing it into your Gitlab storage, and the other components in your DevOps will determine how to build the pushed projects (e.g., via using Maven, Gradle, or Sbt), and once built, tests will run and test coverage and test passing information will be collected. Then, the Understand API calls will be used by some component in your DevOps to process the content of the obtained software applications and to construct the dependency graphs and output a report about modules that have the most dependencies. All kinds of variations are possible here - use your imagination!
In this homework, you will use the Github Developer API not only to stream open-source software projects but also metainformation about them to determine various statistics and to recommend actions to github developers, for example, to recommend what components of the open-source project should be re-tested based on the diff or the patch created as a result of the last code push. Git patches are metadata files that describe how to apply changes to the source code of the previous version of the application to switch to the next consecutive version of the same application. For example, the following lines in a patch "- int a = 2" and "+ float b = 3" would mean that the declaration and initialization of the variable a in some Java file is replaced with the declaration and initalization of the variable b, hence + and - signs in the beginning of the lines. You can find more information in various sources (e.g., https://git-scm.com/docs/git-format-patch).
Please make sure that you were already added as a member of CS_540_2018 team in Bitbucket. Separate repositories will be created for each of your homeworks and for the course project. You will find a corresponding entry for this homework. You will fork this repository and your fork will be private, no one else besides you, your teammates and your course instructor will have access to your fork. Please remember to grant a read access to your repository to your instructor. You can commit and push your code as many times as you want. Your code will not be visible and it should not be visible to other students, except for your teammates. When you push it, your instructor will see you code in your separate private fork. Making your fork public or inviting other students to join your fork will result in losing your grade. For grading, only the latest push timed before the deadline will be considered. If you push after the deadline, your grade for the homework will be zero. For more information about using git and bitbucket specifically, please use this link as the starting point https://confluence.atlassian.com/bitbucket/bitbucket-cloud-documentation-home-221448814.html.
For an additional bonus (up to 5%!) you can integrate other Github data into your project. Your ideas and creativity are highly welcome and will be rewarded! For example, you can obtain issues for each pulled software project and you can attempt to link these issues to specific patches. Or you can link reactions to specific commits and rank program modules w.r.t. reactions types and commits associated with these modules. You can create a separate database into which you can save the attributes of the pulled repos and bug reports. In short, your additional bonus will be based on how you connect various sources of information, not on simply downloading bits and pieces of information. Let your imagination fly!
Even though this is a individual homework, it can be done collaboratively. You are allowed to form groups with up to three teammates. If you want to work alone, it is perfectly fine. Logistically, one of you will create a private fork and will invite one or two of her classmates with the write access to your fork. You should be careful - once you form a group and write and submit code, you cannot start dividing your work and claim you did most of the work. Your forkmates may turn out to be freeloaders and you will be screwed. Be very careful and make sure that you trust your classmates before forming your group. I cannot and I will not resolve your internal group conflicts. Your submission will include the names of all of your forkmates and you will receive the same grade for this homework. Working in a group will be an excellent opportunity for you to explore branching in git, merging, and resolving semantic conflicts when merging your code changes. Don't pass on this opportunity!
I allow you to post questions and replies, statements, comments, discussion, etc. on Piazza. Remember that you cannot share your code and your solutions, but you can ask and advise others using Piazza on where resources and sample programs can be found on the internet, how to resolve dependencies and configuration issues, and how to design the logic of the algorithms and the workflows. Yet, your implementation should be your own or your team's and you cannot share it with the entire class. Alternatively, you cannot copy and paste someone else's implementation and put your name on it. Your submissions will be checked for plagiarism. When posting question and answers on Piazza, please select the appropriate folder, i.e., hw1 to ensure that all discussion threads can be easily located.
Submission deadline: Saturday, March 8 at 7PM CST. Your submission will include your source code, detailed documentation on all aspects of the installation and configuration of your solution, the SBT build configuration, the README.md file in the root directory that contains the description of your implementation, how to compile and run it using SBT, and what are the limitations of your implementation. Please follow this naming convention while submitting your work : "Firstname_Lastname_hw1", so that we can easily recognize your submission. Those who work in groups can use longer names: "Firstname1_Lastname1_Firstname2_Lastname2_Firstname3_Lastname3_hw1". I repeat, please make sure that you will give me read access to your private forked repository.
THE INSTRUCTOR WILL NOT ANSWER ANY REQUESTS FROM STUDENTS STARTING 7PM THE NIGHT BEFORE THE SUBMISSION DEADLINE.
Evaluation criteria: * the maximum grade for this homework is 15%. Points are subtracted from this maximum grade: for example, saying that 2% is lost if some requirement is not completed means that the resulting grade will be 15%-2% => 13%;
no comments or insufficient comments: up to 10% lost;
no unit and integration tests: up to 10% lost;
code does not compile or it crashes without completing the core functionality: up to 10% lost;
the documentation is missing or insufficient to understand how to compile and run your program: up to 10% lost;
only a subset of your functionality works: up to 10% lost;
the minimum grade for this homework cannot be less than zero.