Articles data model and API changes (#1903)

* [WIP] Sort explainers on GraphQL API

* Adding filters to explainers

* Explainers data model and API updates (#1901)

In order to list explainers on the frontend side, some changes were needed on the backend.

- Title should not be mandatory at the API level but at the model level (and description as well)
- Store tags in the Explainer model and keep them in sync with TagText instances
- Adding filters, counter, sorting and pagination to the TeamType.articles connection

Reference: CV2-4500.

* Updates for fact-check data model and API (#1904)

Some changes for fact-checks data model and API:

* Adding tags
* Filters and sort

Reference: CV2-4145

* CV2-4665: index report information in fact check (#1909)

* CV2-4665: index report information in fact check

* CV2-4665: fix tests

* CV2-4665: migrate fact-check tags and allow filter by report informations(publisher, rating and report status)

* CV2-4665: apply PR comments

* CV2-4665: add more tests

* CV2-4665: filter by report information

* Some changes to articles API and data model (#1933)

Some changes to articles API and data model:

- It should be possible to create `ClaimDescription` without a `ProjectMedia`
- Adding a `team_id` field to `ClaimDescription`
- Expose `ClaimDescription.project_media_id` in GraphQL mutations
- Adding a new many-to-many relationship between explainers and items, through a new join model `ExplainItem`, and respective GraphQL type and mutations
- Expose `FactCheck.rating` in GraphQL mutations
- New GraphQL fields for `ProjectMedia`: `fact_check` and `explainers`
- New filters for `Team.explainers` GraphQL connection: `standalone` and `text`

References: CV2-4441, CV2-4626 and CV2-4627.

* Small refactoring

* Fixing two things reported by frontend team

* List standalone fact-checks

* Fixing search by text for explainers

* Ticket CV2-4889: Expose number of articles for an item in GraphQL (#1949)

* Add fact_check_id field to ProjectMediaType

* Adding field `explainer_items` to `ProjectMediaType`

* Exposing fact-check report_status in GraphQL

* Reverting changes to schema

* Adding unique index to explainer_items

* Adding unique index to explainer_items

* Fixing language validation for fact-check

* Add "imported" field to fact-checks (#1951)

Adding an "imported" field to fact-checks. It's automatically set as "true" for fact-checks created by bots. This PR includes:

- Database migration
- Business logic (set "imported" as "true" for fact-checks created by bots, automatically)
- Unit tests
- GraphQL API, including filter to TeamType.articles
- Rake task to update existing fact-checks

Reference: CV2-4882.

* Fixing test

* CV2-4879: add rake task to set team_id for ClaimDescription (#1954)

* CV2-4901 fact check article list not displaying rating (#1952)

* CV2-4901: Sync status value with fact-check rating

* CV2-4901: apply PR comment

* CV2-4901: fix tests

* Always set claim description team based on project media

* Always set claim description team based on project media

* 4880 – Seeds Script: Create standalone claim descriptions and fact-checks (#1956)

This creates both standalone claim descriptions and standalone claim descriptions with fact checks. Though only the second is visible in the UI. We also:

- verify the fact checks with a random status ('undetermined', 'not_applicable', 'in_progress', 'verified' or 'false')
- add a link to half of the standalone fact checks

Note: I also had to update how we were verifying and publishing the fact-checks that are related to a project media. It was breaking since now we can have claim_descriptions with nil project_media_id. (check-api/db/seeds.rb: line 333)

References: 4880
PR: 1956

* Set initial rating for fact-check

* Fixing test

* Fix

* Return team for fact check mutations

* Return total number of articles (regardless the type)

* Adding missing test

---------

Co-authored-by: Brian Fleming <[email protected]>
Co-authored-by: Sawy <[email protected]>
Co-authored-by: Alexandre Amorim <[email protected]>
Co-authored-by: Manu Vasconcelos <[email protected]>

app/graph/mutations/claim_description_mutations.rb

@@ -8,13 +8,12 @@ module SharedCreateAndUpdateFields
     included do
       argument :description, GraphQL::Types::String, required: false
       argument :context, GraphQL::Types::String, required: false, as: :claim_context
+      argument :project_media_id, GraphQL::Types::Int, required: false, camelize: false
     end
   end
 
   class Create < Mutations::CreateMutation
     include SharedCreateAndUpdateFields
-
-    argument :project_media_id, GraphQL::Types::Int, required: true, camelize: false
   end
 
   class Update < Mutations::UpdateMutation

app/graph/mutations/explainer_item_mutations.rb

@@ -0,0 +1,11 @@
+module ExplainerItemMutations
+  MUTATION_TARGET = 'explainer_item'.freeze
+  PARENTS = ['explainer', 'project_media'].freeze
+
+  class Create < Mutations::CreateMutation
+    argument :explainer_id, GraphQL::Types::Int, required: true
+    argument :project_media_id, GraphQL::Types::Int, required: true
+  end
+
+  class Destroy < Mutations::DestroyMutation; end
+end

app/graph/mutations/explainer_mutations.rb

@@ -6,10 +6,11 @@ module SharedCreateAndUpdateFields
     extend ActiveSupport::Concern
 
     included do
-      argument :title, GraphQL::Types::String, required: true
+      argument :title, GraphQL::Types::String, required: false
       argument :description, GraphQL::Types::String, required: false
       argument :url, GraphQL::Types::String, required: false
       argument :language, GraphQL::Types::String, required: false
+      argument :tags, [GraphQL::Types::String, null: true], required: false
     end
   end
 

app/graph/mutations/fact_check_mutations.rb

@@ -1,13 +1,15 @@
 module FactCheckMutations
   MUTATION_TARGET = 'fact_check'.freeze
-  PARENTS = ['claim_description'].freeze
+  PARENTS = ['claim_description', 'team'].freeze
 
   module SharedCreateAndUpdateFields
     extend ActiveSupport::Concern
 
     included do
       argument :url, GraphQL::Types::String, required: false
       argument :language, GraphQL::Types::String, required: false
+      argument :tags, [GraphQL::Types::String, null: true], required: false
+      argument :rating, GraphQL::Types::String, required: false
     end
   end
 

app/graph/types/article_union.rb

@@ -2,5 +2,6 @@ class ArticleUnion < BaseUnion
   description 'A union type of all article types we can handle'
   possible_types(
     ExplainerType,
+    FactCheckType,
   )
 end

app/graph/types/claim_description_type.rb

@@ -14,7 +14,7 @@ class ClaimDescriptionType < DefaultObject
 
   def fact_check(report_status: nil)
     ability = context[:ability] || Ability.new
-    status = object.project_media.report_status
+    status = object.project_media&.report_status
     can_read = ability.can?(:read, object) || status == 'published'
     (can_read && (!report_status || status == report_status)) ? object.fact_check : nil
   end

app/graph/types/explainer_item_type.rb

@@ -0,0 +1,10 @@
+class ExplainerItemType < DefaultObject
+  description 'Explainer item type'
+
+  implements GraphQL::Types::Relay::Node
+
+  field :explainer_id, GraphQL::Types::Int, null: false
+  field :project_media_id, GraphQL::Types::Int, null: false
+  field :explainer, ExplainerType, null: false
+  field :project_media, ProjectMediaType, null: false
+end

app/graph/types/explainer_type.rb

@@ -1,5 +1,5 @@
 class ExplainerType < DefaultObject
-  description "Explainer type"
+  description 'Explainer type'
 
   implements GraphQL::Types::Relay::Node
 
@@ -12,10 +12,5 @@ class ExplainerType < DefaultObject
   field :team_id, GraphQL::Types::Int, null: true
   field :user, UserType, null: true
   field :team, PublicTeamType, null: true
-
-  field :tags, TagType.connection_type, null: true
-
-  def tags
-    Tag.where(annotation_type: 'tag', annotated_type: object.class.name, annotated_id: object.id)
-  end
+  field :tags, [GraphQL::Types::String, null: true], null: true
 end

app/graph/types/fact_check_type.rb

@@ -10,4 +10,8 @@ class FactCheckType < DefaultObject
   field :language, GraphQL::Types::String, null: true
   field :user, UserType, null: true
   field :claim_description, ClaimDescriptionType, null: true
+  field :tags, [GraphQL::Types::String, null: true], null: true
+  field :rating, GraphQL::Types::String, null: true
+  field :imported, GraphQL::Types::Boolean, null: true
+  field :report_status, GraphQL::Types::String, null: true
 end

app/graph/types/mutation_type.rb

@@ -149,4 +149,7 @@ class MutationType < BaseObject
 
   field :createApiKey, mutation: ApiKeyMutations::Create
   field :destroyApiKey, mutation: ApiKeyMutations::Destroy
+
+  field :createExplainerItem, mutation: ExplainerItemMutations::Create
+  field :destroyExplainerItem, mutation: ExplainerItemMutations::Destroy
 end

app/graph/types/project_media_type.rb

@@ -7,6 +7,7 @@ class ProjectMediaType < DefaultObject
 
   field :media_id, GraphQL::Types::Int, null: true
   field :user_id, GraphQL::Types::Int, null: true
+  field :fact_check_id, GraphQL::Types::Int, null: true
   field :url, GraphQL::Types::String, null: true
   field :full_url, GraphQL::Types::String, null: true
   field :quote, GraphQL::Types::String, null: true
@@ -370,4 +371,18 @@ def is_secondary
   field :similar_items, ProjectMediaType.connection_type, null: true
 
   field :media_slug, GraphQL::Types::String, null: true
+
+  field :fact_check, FactCheckType, null: true
+
+  field :explainers, ExplainerType.connection_type, null: true
+
+  field :explainer_items, ExplainerItemType.connection_type, null: true
+
+  field :articles_count, GraphQL::Types::Int, null: true
+
+  def articles_count
+    count = object.explainers.count
+    count += 1 if object.fact_check
+    count
+  end
 end

app/graph/types/team_type.rb

@@ -288,10 +288,64 @@ def tipline_messages(uid:)
 
   field :articles, ::ArticleUnion.connection_type, null: true do
     argument :article_type, GraphQL::Types::String, required: true, camelize: false
-  end
 
-  def articles(article_type:)
-    object.explainers if article_type == 'explainer'
+    # Sort and pagination
+    argument :offset, GraphQL::Types::Int, required: false, default_value: 0
+    argument :sort, GraphQL::Types::String, required: false, default_value: 'title'
+    argument :sort_type, GraphQL::Types::String, required: false, camelize: false, default_value: 'ASC'
+
+    # Filters
+    argument :user_ids, [GraphQL::Types::Int, null: true], required: false, camelize: false
+    argument :tags, [GraphQL::Types::String, null: true], required: false, camelize: false
+    argument :language, [GraphQL::Types::String, null: true], required: false, camelize: false
+    argument :updated_at, GraphQL::Types::String, required: false, camelize: false # JSON
+    argument :text, GraphQL::Types::String, required: false, camelize: false # Search by text
+    argument :standalone, GraphQL::Types::Boolean, required: false, camelize: false # Not applied to any item (fact-checks only)
+    argument :publisher_ids, [GraphQL::Types::Int, null: true], required: false, camelize: false
+    argument :report_status, [GraphQL::Types::String, null: true], required: false, camelize: false
+    argument :rating, [GraphQL::Types::String, null: true], required: false, camelize: false
+    argument :imported, GraphQL::Types::Boolean, required: false, camelize: false # Only for fact-checks
+  end
+
+  def articles(**args)
+    sort = args[:sort].to_s
+    order = [:title, :language, :updated_at, :id].include?(sort.downcase.to_sym) ? sort.downcase.to_sym : :title
+    order_type = args[:sort_type].to_s.downcase.to_sym == :desc ? :desc : :asc
+    articles = Explainer.none
+    if args[:article_type] == 'explainer'
+      articles = object.filtered_explainers(args)
+    elsif args[:article_type] == 'fact-check'
+      articles = object.filtered_fact_checks(args)
+    end
+    articles.offset(args[:offset].to_i).order(order => order_type)
+  end
+
+  field :articles_count, GraphQL::Types::Int, null: true do
+    argument :article_type, GraphQL::Types::String, required: false, camelize: false
+
+    # Filters
+    argument :user_ids, [GraphQL::Types::Int, null: true], required: false, camelize: false
+    argument :tags, [GraphQL::Types::String, null: true], required: false, camelize: false
+    argument :language, [GraphQL::Types::String, null: true], required: false, camelize: false
+    argument :updated_at, GraphQL::Types::String, required: false, camelize: false # JSON
+    argument :text, GraphQL::Types::String, required: false, camelize: false # Search by text
+    argument :standalone, GraphQL::Types::Boolean, required: false, camelize: false # Not applied to any item (fact-checks only)
+    argument :publisher_ids, [GraphQL::Types::Int, null: true], required: false, camelize: false
+    argument :report_status, [GraphQL::Types::String, null: true], required: false, camelize: false
+    argument :rating, [GraphQL::Types::String, null: true], required: false, camelize: false
+    argument :imported, GraphQL::Types::Boolean, required: false, camelize: false # Only for fact-checks
+  end
+
+  def articles_count(**args)
+    count = nil
+    if args[:article_type] == 'explainer'
+      count = object.filtered_explainers(args).count
+    elsif args[:article_type] == 'fact-check'
+      count = object.filtered_fact_checks(args).count
+    elsif args[:article_type].blank?
+      count = object.filtered_explainers(args).count + object.filtered_fact_checks(args).count
+    end
+    count
   end
 
   field :api_key, ApiKeyType, null: true do

app/models/ability.rb

@@ -170,9 +170,10 @@ def collaborator_perms
       v_obj = obj.item_type.constantize.find(obj.item_id) if obj.item_type == 'ProjectMedia'
       !v_obj.nil? and v_obj.team_id == @context_team.id and v_obj.media.user_id = @user.id
     end
-    can [:create, :update, :read, :destroy], FactCheck, { claim_description: { project_media: { team_id: @context_team.id } } }
+    can [:create, :update, :read, :destroy], FactCheck, { claim_description: { team_id: @context_team.id } }
     can [:create, :update, :read, :destroy], Explainer, team_id: @context_team.id
-    can [:create, :update, :read], ClaimDescription, { project_media: { team_id: @context_team.id } }
+    can [:create, :update, :read], ClaimDescription, { team_id: @context_team.id }
+    can [:create, :update, :read, :destroy], ExplainerItem, { project_media: { team_id: @context_team.id } }
   end
 
   def bot_permissions

app/models/claim_description.rb

@@ -1,13 +1,16 @@
 class ClaimDescription < ApplicationRecord
   include Article
 
-  belongs_to :project_media
+  before_validation :set_team, on: :create
+  belongs_to :project_media, optional: true
+  belongs_to :team
   has_one :fact_check, dependent: :destroy
 
   accepts_nested_attributes_for :fact_check, reject_if: proc { |attributes| attributes['summary'].blank? }
 
-  validates_presence_of :project_media
-  validates_uniqueness_of :project_media_id
+  validates_presence_of :team
+  validates_uniqueness_of :project_media_id, allow_nil: true
+  after_commit :update_fact_check, on: [:update]
 
   # To avoid GraphQL conflict with name `context`
   alias_attribute :claim_context, :context
@@ -32,4 +35,19 @@ def article_elasticsearch_data(action = 'create_or_update')
     }
     self.index_in_elasticsearch(data)
   end
+
+  private
+
+  def set_team
+    team = (self.project_media&.team || Team.current)
+    self.team = team unless team.nil?
+  end
+
+  def update_fact_check
+    fact_check = self.fact_check
+    if fact_check && self.project_media_id
+      fact_check.updated_at = Time.now
+      fact_check.save!
+    end
+  end
 end

app/models/concerns/article.rb

@@ -15,6 +15,7 @@ module Article
 
     after_commit :update_elasticsearch_data, :send_to_alegre, :notify_bots, on: [:create, :update]
     after_commit :destroy_elasticsearch_data, on: :destroy
+    after_save :create_tag_texts_if_needed
   end
 
   def text_fields
@@ -46,14 +47,15 @@ def notify_bots
       'ClaimDescription' => 'save_claim_description',
       'FactCheck' => 'save_fact_check'
     }[self.class.name]
-    BotUser.enqueue_event(event, self.project_media.team_id, self)
+    BotUser.enqueue_event(event, self.project_media.team_id, self) unless self.project_media.nil?
   end
 
   protected
 
   def index_in_elasticsearch(data)
     # touch project media to update `updated_at` date
     pm = self.project_media
+    return if pm.nil?
     pm = ProjectMedia.find_by_id(pm.id)
     unless pm.nil?
       updated_at = Time.now
@@ -64,9 +66,25 @@ def index_in_elasticsearch(data)
     end
   end
 
+  def create_tag_texts_if_needed
+    self.class.delay.create_tag_texts_if_needed(self.team_id, self.tags) if self.respond_to?(:tags) && !self.tags.blank?
+  end
+
   module ClassMethods
+    def create_tag_texts_if_needed(team_id, tags)
+      tags.each do |tag|
+        next if TagText.where(text: tag, team_id: team_id).exists?
+        tag_text = TagText.new
+        tag_text.text = tag
+        tag_text.team_id = team_id
+        tag_text.skip_check_ability = true
+        tag_text.save!
+      end
+    end
+
     def send_to_alegre(id)
       obj = self.find_by_id(id)
+      return if obj.project_media.nil?
       obj.text_fields.each do |field|
         ::Bot::Alegre.send_field_to_similarity_index(obj.project_media, field)
       end unless obj.nil?

app/models/concerns/project_media_associations.rb

@@ -21,6 +21,8 @@ module ProjectMediaAssociations
     has_one :claim_description, dependent: :destroy
     belongs_to :source, optional: true
     has_many :tipline_requests, as: :associated
+    has_many :explainer_items
+    has_many :explainers, through: :explainer_items
     has_annotations
   end
 end

app/models/concerns/project_media_cached_fields.rb

@@ -73,7 +73,8 @@ def title_or_description_update
       model: FactCheck,
       affected_ids: proc { |fc| [fc.claim_description.project_media] },
       events: {
-        save: :recalculate
+        save: :recalculate,
+        destroy: :recalculate
       }
     }
 
@@ -176,6 +177,11 @@ def title_or_description_update
         }
       ]
 
+    cached_field :fact_check_id,
+      start_as: nil,
+      recalculate: :recalculate_fact_check_id,
+      update_on: [FACT_CHECK_EVENT]
+
     cached_field :fact_check_title,
       start_as: nil,
       recalculate: :recalculate_fact_check_title,
@@ -540,6 +546,10 @@ def recalculate_last_seen
       [v1, v2].max.to_i
     end
 
+    def recalculate_fact_check_id
+      self.claim_description&.fact_check&.id
+    end
+
     def recalculate_fact_check_title
       self.claim_description&.fact_check&.title
     end

app/models/concerns/project_media_getters.rb

@@ -209,4 +209,8 @@ def get_creator_name
   def team_avatar
     self.team.avatar
   end
+
+  def fact_check
+    self.claim_description&.fact_check
+  end
 end

app/models/concerns/team_associations.rb

@@ -116,4 +116,8 @@ def check_search_unconfirmed
   def check_search_spam
     check_search_filter({ 'archived' => CheckArchivedFlags::FlagCodes::SPAM })
   end
+
+  def fact_checks
+    FactCheck.joins(:claim_description).where('claim_descriptions.team_id' => self.id)
+  end
 end

app/models/explainer.rb

@@ -4,10 +4,12 @@ class Explainer < ApplicationRecord
   belongs_to :team
 
   has_annotations
+  has_many :explainer_items
+  has_many :project_medias, through: :explainer_items
 
   before_validation :set_team
   validates_format_of :url, with: URI.regexp, allow_blank: true, allow_nil: true
-  validates_presence_of :team
+  validates_presence_of :team, :title, :description
   validate :language_in_allowed_values, unless: proc { |e| e.language.blank? }
 
   def notify_bots