Your syndication has been served

This is a small one, but worthy of note. We took away the rss feeds from Node.

• https://api.gregory-ms.com/feed/latest/articles/
• https://api.gregory-ms.com/feed/latest/trials/
• https://api.gregory-ms.com/feed/machine-learning/ Shows the latest articles that Gregory believes to be relevant.

We also managed to include the European Clinical Trial Register as a source of information.

What's Changed

• bypass SSL check to parse the European Clinical Trial Register feed by @brunoamaral in #86
• Add rss feeds via Django by @brunoamaral in #87

Full Changelog: v8...v8.1

v8 - Building a proper API

This won't hit as hard as Smith but it does pack a punch.
And and I have a hunch this release will rock.

The API is now hosted at https://api.gregory-ms.com/ and offers some options for authentication, new methods, and much more information about each piece of content. Articles, Trials, and Sources.

The search by keyword is somewhat new as it now only uses get requests and accepts regular expressions. The javascript that shows related articles on the site is now broken, but I'll get around to fix it sometime next week. Same as the documentation for developers, that has become more outdated than the yellow pages.

Life has been a bit demanding lately, so if you can, please take a look around and let me know if you find any issues.

What's Changed

• Move the API to Django Rest Framework by @brunoamaral in #77
• remove extra Z in date time, fixes #56 by @brunoamaral in #70
• adds doi number to database, solves #72 by @brunoamaral in #73
• add missing api endpoint for machine learning model by @brunoamaral in #78
• adjust to use the future API endpoint by @brunoamaral in #79
• make sure summary is over 360 characters by @brunoamaral in #82
• add missing condition to view by @brunoamaral in #83
• seems to fix #81 by @brunoamaral in #84

Full Changelog: v7...v8

v7 - The grown up edition

Alright, put down that coffee, this is a though one. You'll see, come here and sit with me.

I had to break a lot of things.

1. Postgres (PG) Database
2. Node Red flows
3. New way to fetch RSS feeds
4. New admin option
5. Metabase module

Database

Gregory was using a SQLite database, which is ok for most uses and was enough for the first year. The data was moved to a new Postgres database. This solves issues we were having with incoherent dates and allows Metabase to run a better analysis on everything involving a date and time. Same goes for True and False values, which SQLite considered to be either 0 and 1 and made some queries feel weird.

One change was adding some new fields to the table of sources, these now include the method used to fetch new articles. More on that up ahead.

Node Red flows

New database meant reviewing the flows that search and retrieve new articles, making them use PG. During this, I cleaned up some nodes and flows that didn't make much sense. Sorry for not documenting this, but either way, Gregory is meant to mostly use your flows and not what I believe is right.

New way to fetch rss feeds

On the python-ml directory there is now a feedreader.py script that will query the database for a list of sources that use RSS, fetch new articles and add them to the database. This may feel like a nuisance, but it's necessary to move towards a system where you add sources through an admin panel instead of coding a Node Red flow for each one.

New admin option

Sometimes you need to edit information on the articles, sources or trials. To make this easier I have added a Django install to the docker-compose.yaml file.

When running this for the first time, *remember to execute python manage.py migrate and then python manage.py createsuperuser so that it will add the necessary tables and a user account.

This is available at http://localhost:8000 and you will have to bind it with your preferred reverse proxy.

Metabase module

I broke this harder than my last girlfriend broke my heart.

It didn't make sense to have another PG container running just for Metabase. Instead, it was added to the new one that is holding the articles and clinical trials. We keep things separate by using two different databases. Should have done the same for the Django install but didn't know how to and had to pick priorities.

This means that the charts in the observatory section of the website have all gone to hell and will need to brought back. User accounts will also need to be recreated.

That's all, and that's a lot

This was a lot of work and it will break all other installs of Gregory, I'm sorry about that. Hopefully you will agree it was a necessary step towards a system that is easier to customize and adapt to other conditions other than Multiple Sclerosis.

I'd like to thank @malduarte and @chbm who gave me a hand in migrating to PG. It would have been a lot harder without their help.

Any questions, comments or suggestions are more than welcome. Meanwhile I am just going to take it easy for rest of the week and calmly make sure everything is working as intended.

falls down in Portuguese

What's Changed

I messed up the auto-generated release notes, so you'll want to check the v7 milestone.

v6.1.2

The scikit package was updated so if you are running previous versions, this requires you to retrain the machine learning algorithm. This means running script 1 and 2 in the python-ml/ directory.

What's Changed

• 6.0.1 - recover flows by @brunoamaral in #47
• Make sure we have all the packages to run Scikit by @brunoamaral in #49
• Update database schema and add information on new database column
• Cleaned up code
• Improved setup instructions
• Updated flows
• Removed unused directory and references to it in build.py

Full Changelog: v6...v6.1.2

v6 - Spacy meets Node-Red

Ok, this is a doozy.

This release updates on four fronts.

1. Node-RED

Gregory is now running Node-RED v2.1.4, this is big all by itself. Visit @node-red for more details.

This fixes #37

2. Docker image

There were a few changes to the Dockerfile, and you no longer need to build the container locally. The image for Gregory is available on the Docker Hub and is referenced on the docker-compose.yaml file.

The container is now running Python 3.7.3.

3. Setup script was improved

There is now a .env file that will hold all the settings you need for the docker-compose.yaml and the build.py script. In addition, the setup.py file will make sure you create the needed docker network and fetch the current version of the docker image.

4. More ArtificiaI Intelligence and Machine Learning

The new docker image was built with the Spacy python module by @explosion . This allows Gregory to incorporate Natural Language Processing (NLP) and other features into the flows that fetch and analyse search results. Soon, it will also allow the site to be built a little bit faster, by storing the NLP results in the database.

For reference, the site currently takes about 5 minutes to built and part of this includes the NLP processing of titles to identify noun phrases.

This fixes #32

Thank you
— Bruno

v5 - Metabase and refactored flows

The most significant change for this release is the optional docker-compose.yaml file that allows running Metabase alongside Gregory.

Metabase is available at https://www.metabase.com/ and @metabase.

Content pages were updated, and algolia was removed from the search page in favor of a Metabase dashboard that allows for less dependencies and is always up to date.

v4 - Search page

There is now a search page available https://gregory-ms.com/search/

This is using https://algolia.com and in future versions we need to include updating the index automatically.

Right now, I am updating the index with a manual file upload every other day.

v3 - Machine Learning

Gregory just got a little bit smarter with the help of @antoniolopes.

This releases includes a lot of changes to the frontend and to node-red's flows, biggest one being that we now tag articles on the homepage with a machine learning algorithm.

There are two Machine Learning models available:

• Gaussian Naive Bayes (GNB)
• Logistic Regression (LR)

Right now, the homepage uses the GNB model because in testing it matched human selection in the best way.

v2.0 - New theme

This is mostly an aesthetic update. The site now uses the Now-UI theme from Creative Tim, with tons of changes to fit the purpose of listing research and clinical trials.

Some highlights of the theme:

• Pagination of the homepage and clinical trials sections
• Thumbnails for when static pages are shared on social media
• When available, abstracts are now listed after the title of the Article/Trial

Changes to the build script:

• We now produce an index.md for each Article or Clinical Trial
• Prints out each stage of the build process

We are in contact with a few stakeholders and hope to improve Gregory with their help.
Thank you to everyone reading this. 🙂

v1.9 - Docker and Machine Learning

This version includes a Dockerfile and docker-compose.yaml to allow anyone to install Gregory with ease.

It also includes a set of Machine Learning scripts and models to make selecting relevant articles easier. These modules still need to be included in the Node-RED Flows.

Special thanks to @antoniolopes and @Madril 🙂