Backend

Server-side Implementation

Crowdcode’s server is implemented on a Node.js platform with the Express.js framework. The advantage of Node server is that it is asynchronous. This means there is only one thread that runs continuously, which does not wait for requests to be responded before moving on to another request, increasing server performance. In this version, there are only two types of microtasks.

1. Implementation – The worker can write code or tests or do both.
2. Review – The worker shall review implementation tasks submitted by another worker. Either accept or reject

Differences with the old version

In the traditional approach

1. Client sends requests
2. The server fetches data from the database
3. Performs operations
4. Stores data back into the database
5. Responds to request

But this is modified in the current implementation of Crowd Microservises, where all the project data related to microtasks is eagerly loaded into the servers’ memory at the first request for a project.

1. Client sends request
2. The server performs the operation on the data memory
3. Responds to client
4. Updates database

There are 4 main services in the server

1. User Services – Methods for user login and logout as well as obtaining user data such as avatar pic, email id, etc.
2. Firebase Service – Methods to create, update, retrieve and delete all the objects in firebase.
3. Microtask Service – Methods to load projects into memory as well as methods to perform generate, submit, fetch and skip microtask operations in the server
4. Deployment Service – Methods to create a microservice from the project and deploy it as a separate app on Heroku.

Microtask Service

1. LoadProject – Loads a project into the memory from firebase, if it does not already exist
2. LoadFunctions – Loads all the functions that are yet to be implemented
3. LoadTests – Loads all the tests for each function that is present in the memory
4. LoadState – If the project was previously being implemented, it loads the state of the project into memory
5. LoadMicrotasks – Loads all the microtasks, that are listed in the state or generates the tasks
6. GenerateImplementationTask – Generates an implementation microtask for a function
7. GenerateReviewTask – Generates a review microtask based on the implementation task
8. SubmitImplementationTask – Stores the user submission in microtask and calls for the generation of a review task.
9. SubmitReviewTask – Stores the user review and rating. Based on which the function is updated/not updated. Calls for generation of implementation task if the function is not completed. If the function is complete, it is removed from memory and an implementation task is generated for the next incomplete function.
10. FetchMicrotask – Returns a microtask. Review tasks always have priority over implementation. The method checks if the user requesting already had a task assigned, in which case it returns the same task again. It triggers a timer for the task to be auto-submitted after the interval
11. SkipMicrotask – Add the users' current microtask to the skipped tasks list for that user. Add the microtask back into its respective queue and calls fetch microtask. If there are no new tasks in the queue and there is more than one task in the skipped task list, it clears the skipped tasks list and then calls for fetch microtask
12. UnassignLockedMicrotask – If the task is not submitted even after the allocated time, this changes its status back to unassigned and creates a new microtask. The previous task is flagged as submitted. The assigned task for the user is also changed to null.

Firebase Service

1. All interactions are logged in the database in the \histroy\events via firebase.createEvent method

Implementation decisions

The server holds data in the form of maps (shown below) with keys generated by firebase and values being the JSON objects. Some queues are implemented using stacks (as queues are not available in JavaScript). Below is a list of all the objects held by the server in its memory.

1. Projects Map – holds all the projects a. (Project Id, Project Map)
2. Project Map – Hold individual project a. (“functions”, Functions Map) b. (“tests”, Tests Map) c. (“microtasks”, Microtasks Map) d. (“workers”, Workers Map) e. (“ImplementationQ”, Implementation Queue Array) f. (“reviewQ”, Review Queue Array) g. (“inProgressQ”, In Progress Map)
3. Functions Map a. (function id, function object)
4. Tests Map a. (test id, test object)
5. Microtasks Map a. (microtask id, microtask object)
6. Implementation Queue Array a. Implementation Microtask Ids
7. Review Queue Array a. Review Microtasks Ids
8. In Progress Queue Map a. (microtask id, Microtask Map)
9. Microtask Map a. (“worker”, Worker Id) b. (“assigned_time”, Timestamp)
10. Workers Map a. (“assigned”, Assigned task Map) b. (“skipped” Skipped Tasks Array) c. (“completed”, Completed tasks Array)
11. Assigned task Map a. (“id”, Microtask Id) b. (“type”, Microtask Type) c. (“fetch_time”, Timestamp)
12. Skipped Tasks Array a. Skipped Microtask Ids
13. Completed Tasks Array a. Completed Microtasks Ids