Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/import ski trail maintenance history #387

Open
wants to merge 69 commits into
base: develop
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
4eeb501
Copied street maintenance model
juuso-j Sep 17, 2024
6b8cf19
Add maintenance to installed apps
juuso-j Sep 17, 2024
8bf546b
Delete as move to maintenace application
juuso-j Sep 17, 2024
7818d81
Fix import
juuso-j Sep 17, 2024
14e1c53
Delete, moved to maintenance app
juuso-j Sep 17, 2024
2a8ae44
Copy tests from street_maintenance app
juuso-j Sep 17, 2024
f61b526
Delete, moved to maintenance applicaiton
juuso-j Sep 18, 2024
1534e76
Delete, moved to maintenance app
juuso-j Sep 18, 2024
f10ca1c
Change street_maintenance to maintenance in DOC_ENDPOINTS
juuso-j Sep 18, 2024
1ff9f2a
Add tests from street_maintenance application
juuso-j Sep 18, 2024
8a6f9e2
Add __init__.py
juuso-j Sep 18, 2024
bd1d664
Add __init__.py
juuso-j Sep 18, 2024
9e4f796
Add tasks.py from street_maintenance application
juuso-j Sep 18, 2024
d52b36e
Add maintenance AppConfig
juuso-j Sep 18, 2024
1f34e1d
Delete all street_maintenance application tables
juuso-j Sep 18, 2024
27ba44b
Add constants.py from street_maintenance
juuso-j Sep 18, 2024
6111a72
Add models from street_maintenance application
juuso-j Sep 18, 2024
fec6ada
Add API from street_maintenace application
juuso-j Sep 18, 2024
0e261ba
Add admin from street_maintenance
juuso-j Sep 18, 2024
e59ba1b
Add street_maintenance urls for backward compatibility
juuso-j Sep 18, 2024
136d5fd
Add maintenance urls
juuso-j Sep 18, 2024
658c34a
Replace street_maintenance logger settings with maintenance
juuso-j Sep 18, 2024
875ba9e
Add README from street_maintenance
juuso-j Sep 18, 2024
76eb781
Add utils.py from street_maintenance applicaiton
juuso-j Sep 18, 2024
177ad0e
Add management commands from street_maintenance application
juuso-j Sep 18, 2024
c728fec
Add specifications
juuso-j Sep 18, 2024
b6d615b
Delete, as moved to maintenance application
juuso-j Sep 18, 2024
a75d5ba
Fix maintenance urls prefix
juuso-j Sep 19, 2024
23670b8
Merge branch 'feature/replace-street-maintenance-app-with-maintenance…
juuso-j Sep 19, 2024
121e21d
Rename file
juuso-j Sep 24, 2024
1993b3a
Add function get_data_layer
juuso-j Sep 25, 2024
5d2f976
Add importer for ski trails
juuso-j Sep 25, 2024
1b707db
Add task to import ski trails
juuso-j Sep 25, 2024
49dd07b
Add fixture data for testing importing of ski trails
juuso-j Sep 25, 2024
f9f39ac
Add ski trails importer tests
juuso-j Sep 25, 2024
145d5cb
Add function to get fixture GDAL data layer
juuso-j Sep 26, 2024
c77fcaf
Add UnitMaintenanceGeometry fixture
juuso-j Sep 26, 2024
1782b2d
Add models for storing Unit maintenance history
juuso-j Sep 26, 2024
d433930
Add Unit fixture
juuso-j Sep 26, 2024
702ea7e
Add ski trails maintenance history importer
juuso-j Sep 26, 2024
039e6ce
Add ski_trails_maintenance_history_mock_data
juuso-j Sep 26, 2024
8bda877
Add ski trail maintenance history importer tests
juuso-j Sep 26, 2024
f6a1287
Add import_ski_trails_maintenance_history task
juuso-j Sep 26, 2024
73b4e98
Add view for UnitMaintenance model
juuso-j Sep 27, 2024
39df1c9
Add fixture now
juuso-j Sep 27, 2024
249da09
Add units fixture
juuso-j Sep 27, 2024
91345a5
Add unit_maintenance_geometries fixture
juuso-j Sep 27, 2024
a2d2af5
Add SKI_TRAILS_DATE_FIELD_FORMAT constants
juuso-j Sep 27, 2024
3cd24f5
Add UnitMaintenance fixtures
juuso-j Sep 27, 2024
c1017a2
Add viewset for UnitMaintenanceGeometry
juuso-j Sep 27, 2024
12277a4
Add serializers for UnitMaintenance and UnitMaintenanceGeometry
juuso-j Sep 27, 2024
676d8c4
Add tests for UnitMaintenanceViewSet
juuso-j Sep 27, 2024
c905115
Add unit_maintenance/ to DOC_ENDPOINTS
juuso-j Sep 27, 2024
e8bc406
Add UnitMaintenance and UnitMaintenanceGeometry
juuso-j Sep 27, 2024
4d79211
Test UnitMaintenanceGeometryViewSet
juuso-j Sep 27, 2024
7436e3a
Add information about ski trails
juuso-j Sep 27, 2024
cabb95f
Conditionally serialize maintenance information to units
juuso-j Sep 27, 2024
e2ab6e0
Serialize maintance information by default
juuso-j Sep 27, 2024
f3d5d2f
Fix SKITRAIL_TO_UNIT_ID_MAPPINGS
juuso-j Sep 30, 2024
6b15e4a
Change STREET_MAINTENANCE_LOG_LEVEL to MAINTENANCE_LOG_LEVEL
juuso-j Oct 1, 2024
acfc917
Merge branch 'feature/replace-street-maintenance-app-with-maintenance…
juuso-j Oct 1, 2024
9ace48f
Fix comment
juuso-j Oct 1, 2024
b56a5c3
Merge branch 'feature/replace-street-maintenance-app-with-maintenance…
juuso-j Oct 1, 2024
b7558aa
Add SKI_TRAILS_URL environment variable
juuso-j Oct 2, 2024
889932a
Add SKI_TRAILS_MAINTENANCE_HISTORY_URL environment variable
juuso-j Oct 2, 2024
21aeab0
Test ski trails geometry change
juuso-j Oct 2, 2024
028788b
Set unit_maintenance deletion to CASCADE
juuso-j Oct 2, 2024
a6c612f
Remove ueless line of code
juuso-j Oct 2, 2024
161b322
Handle geometry change
juuso-j Oct 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add API from street_maintenace application
juuso-j committed Sep 18, 2024
commit fec6ada47a7174c0e8c6dbe0cf61399eeb7f558c
64 changes: 64 additions & 0 deletions maintenance/api/serializers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
from django.contrib.gis.geos import LineString, Point
from drf_spectacular.types import OpenApiTypes
from drf_spectacular.utils import extend_schema_field
from rest_framework import serializers

from maintenance.models import GeometryHistory, MaintenanceUnit, MaintenanceWork


class GeometryHistorySerializer(serializers.ModelSerializer):
geometry_type = serializers.SerializerMethodField()

class Meta:
model = GeometryHistory
fields = [
"id",
"geometry_type",
"events",
"timestamp",
"provider",
# Removed for permormance issues as it is not currently used
# "geometry",
"coordinates",
]

@extend_schema_field(OpenApiTypes.STR)
def get_geometry_type(self, obj):
return obj.geometry.geom_type


class ActiveEventSerializer(serializers.Serializer):
events = serializers.CharField(max_length=64)


class MaintenanceUnitSerializer(serializers.ModelSerializer):
class Meta:
model = MaintenanceUnit
fields = "__all__"


class MaintenanceWorkSerializer(serializers.ModelSerializer):
provider = serializers.PrimaryKeyRelatedField(
many=False, source="maintenance_unit.provider", read_only=True
)

class Meta:
model = MaintenanceWork
fields = [
"id",
"maintenance_unit",
"provider",
"geometry",
"timestamp",
"events",
"original_event_names",
]

def to_representation(self, obj):
representation = super().to_representation(obj)
if isinstance(obj.geometry, Point):
representation["lat"] = obj.geometry.y
representation["lon"] = obj.geometry.x
elif isinstance(obj.geometry, LineString):
representation["coords"] = obj.geometry.coords
return representation
24 changes: 24 additions & 0 deletions maintenance/api/urls.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
from django.urls import include, path
from rest_framework import routers

from . import views

app_name = "maintenance"

router = routers.DefaultRouter()
router.register("active_events", views.ActiveEventsViewSet, basename="active_events")

router.register(
"maintenance_works", views.MaintenanceWorkViewSet, basename="maintenance_works"
)
router.register(
"maintenance_units", views.MaintenanceUnitViewSet, basename="maintenance_units"
)

router.register(
"geometry_history", views.GeometryHitoryViewSet, basename="geometry_history"
)

urlpatterns = [
path("", include(router.urls), name="maintenance"),
]
196 changes: 196 additions & 0 deletions maintenance/api/views.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
from datetime import datetime

from django.utils.decorators import method_decorator
from django.utils.timezone import make_aware
from django.views.decorators.cache import cache_page
from drf_spectacular.utils import extend_schema, extend_schema_view, OpenApiParameter
from rest_framework import mixins, viewsets
from rest_framework.exceptions import ParseError
from rest_framework.pagination import PageNumberPagination

from maintenance.api.serializers import (
ActiveEventSerializer,
GeometryHistorySerializer,
MaintenanceUnitSerializer,
MaintenanceWorkSerializer,
)
from maintenance.management.commands.constants import (
EVENT_CHOICES,
PROVIDERS,
START_DATE_TIME_FORMAT,
)
from maintenance.models import GeometryHistory, MaintenanceUnit, MaintenanceWork

EXAMPLE_TIME_FORMAT = "YYYY-MM-DD HH:MM:SS"
EXAMPLE_TIME = "2022-09-18 10:00:00"
EVENT_PARAM = OpenApiParameter(
name="event",
location=OpenApiParameter.QUERY,
description=(
"Return objects of given event. "
f'Event choices are: {", ".join(EVENT_CHOICES).lower()}, '
'E.g. "auraus".'
),
required=False,
type=str,
)
PROVIDER_PARAM = OpenApiParameter(
name="provider",
location=OpenApiParameter.QUERY,
description=("Return objects of given provider. " 'E.g. "INFRAROAD".'),
required=False,
type=str,
)
START_DATE_TIME_PARAM = OpenApiParameter(
name="start_date_time",
location=OpenApiParameter.QUERY,
description=(
"Get objects with timestamp newer than the start_date_time. "
f'The format for the timestamp is: {EXAMPLE_TIME_FORMAT}, e.g. "{EXAMPLE_TIME}".'
),
required=False,
type=str,
)


class LargeResultsSetPagination(PageNumberPagination):
"""
Custom pagination class to allow all results in one page.
"""

page_size_query_param = "page_size"
# Works are fetched to the remote data storage on a single page, to prevent
# duplicates.
max_page_size = 200_000


class ActiveEventsViewSet(mixins.ListModelMixin, viewsets.GenericViewSet):
queryset = MaintenanceWork.objects.order_by().distinct("events")
serializer_class = ActiveEventSerializer


_maintenance_works_list_parameters = [
EVENT_PARAM,
PROVIDER_PARAM,
START_DATE_TIME_PARAM,
]


@extend_schema_view(
list=extend_schema(
parameters=_maintenance_works_list_parameters,
description="A MaintenanceWork object is a single work performed by a provider. The geometry can be a point "
"or a linestring. The geometry and timestamp are assigned directly from the source data. The "
"original_event_names contains the names of the events in the source data and the events field is the mapped "
"names. Note, if the geometry is a point, the latitude and longitude will be separately serialized. If the "
"geometry is a linestring a separate list of coordinates will be serialized.",
)
)
class MaintenanceWorkViewSet(viewsets.ReadOnlyModelViewSet):
serializer_class = MaintenanceWorkSerializer
pagination_class = LargeResultsSetPagination

def get_queryset(self):
queryset = MaintenanceWork.objects.all()
filters = self.request.query_params

if "provider" in filters:
provider = filters["provider"].upper()
if provider in PROVIDERS:
queryset = queryset.filter(maintenance_unit__provider=provider)
else:
raise ParseError(f"Providers are: {', '.join(PROVIDERS)}")

if "event" in filters:
queryset = queryset.filter(events__contains=[filters["event"]])
if "start_date_time" in filters:
start_date_time = filters["start_date_time"]
try:
start_date_time = datetime.strptime(
start_date_time, START_DATE_TIME_FORMAT
)
except ValueError:
raise ParseError(
f"'start_date_time' must be in format {EXAMPLE_TIME_FORMAT} elem.g.,'{EXAMPLE_TIME}'"
)

queryset = queryset.filter(timestamp__gte=make_aware(start_date_time))
return queryset

def list(self, request):
queryset = self.get_queryset()
filters = self.request.query_params
if "unit_id" in filters:
unit_id = filters["unit_id"]
queryset = queryset.filter(maintenance_unit__unit_id=unit_id)

page = self.paginate_queryset(queryset)
serializer = self.serializer_class(page, many=True)
return self.get_paginated_response(serializer.data)


@extend_schema_view(
list=extend_schema(
description="MaintananceUnit objets are the entities that creates the MaintenanceWorks. Every MaintenanceWork "
"has a relation to a MaintenanceUnit. The type of the MaintenanceUnit can vary depending on the provider. It "
"can be a machine or an event.",
),
)
class MaintenanceUnitViewSet(viewsets.ReadOnlyModelViewSet):
queryset = MaintenanceUnit.objects.all()
serializer_class = MaintenanceUnitSerializer


_geometry_history_list_parameters = [
PROVIDER_PARAM,
EVENT_PARAM,
START_DATE_TIME_PARAM,
]


@extend_schema_view(
list=extend_schema(
parameters=_geometry_history_list_parameters,
description="GeometryHistory objects are processed from MaintenanceWork objects. MaintenanceWorks with "
"linestrings are validated and if valid a GeometryHistory object is created with the linestring geometry. "
"From MaintenanceWork objects that cointains point data linestrings are generated. The linestrings are "
"generated by comparing the timestamp, event, distance and provider. For every valid generated linestring a "
"GeometryHistory object is created. The coordinates are in SRID 4326.",
)
)
class GeometryHitoryViewSet(viewsets.ReadOnlyModelViewSet):
serializer_class = GeometryHistorySerializer
pagination_class = LargeResultsSetPagination

def get_queryset(self):
queryset = GeometryHistory.objects.all()
filters = self.request.query_params

if "provider" in filters:
provider = filters["provider"].upper()
if provider in PROVIDERS:
queryset = queryset.filter(provider=provider)
else:
raise ParseError(f"Providers are: {', '.join(PROVIDERS)}")

if "event" in filters:
queryset = queryset.filter(events__contains=[filters["event"]])
if "start_date_time" in filters:
start_date_time = filters["start_date_time"]
try:
start_date_time = datetime.strptime(
start_date_time, "%Y-%m-%d %H:%M:%S"
)
except ValueError:
raise ParseError(
f"'start_date_time' must be in format {EXAMPLE_TIME_FORMAT} elem.g.,'{EXAMPLE_TIME}'"
)
queryset = queryset.filter(timestamp__gte=make_aware(start_date_time))
return queryset

@method_decorator(cache_page(60 * 15))
def list(self, request):
queryset = self.get_queryset()
page = self.paginate_queryset(queryset)
serializer = self.serializer_class(page, many=True)
return self.get_paginated_response(serializer.data)