Skip to content

Latest commit

 

History

History
83 lines (62 loc) · 5.27 KB

CHANGELOG.md

File metadata and controls

83 lines (62 loc) · 5.27 KB
Raw

0.3.1

  • Introducing KNN + DataFrame API (PR)
  • Fix wrong ordering when using unique in KNN (PR)
  • Introducing Window Query + DataFrame API (PR)
  • Rename addSPartitioning -> prePartition (PR)
  • Accelerated partitioning (aborted and postponed) (PR)

0.3.0

  • Complete change of the interface (PR). This includes:
    • DataFrame API
    • Extension of the Spark SQL module (new methods + implicits on DataFrame)
    • Metadata are propagated (not just 3D coordinates).
    • Seamless interface with Python
  • Compiling against Apache Spark 2.3.2: fix incompatibilities (PR)
  • Website update: spark-fits version + vulnerability fix (PR)
  • Fix wrong angle definition in cartesian to spherical coordinate change! (PR)

0.2.2

  • Add the conversion from FITS file (single HDU) to parquet (PR)
  • Tools to perform live 3D RDD visualisation (in progress) (PR)
  • Several updates to the Travis CI (PR)
  • Add example dealing with collapse function in pyspark (PR)
  • Deploy pyspark3d with pip (PR)
  • Clarify the use of get_spark_session and load_user_conf (for tests only!) (PR)

0.2.1

  • pyspark3d contains all the features of spark3D (partitioning, operators)
  • Scala reflection support for python using py4j (PR)

0.2.0

  • Add support for Python: pyspark3d (PR).
  • Update the Travis script to run test suites for both scala and python (PR).
  • Switch to docker-based travis to test against ubuntu 16.04 (PR).

0.1.5

  • Allow user to cache the rawRDD while loading data to speed-up re-partitioning. (PR)
  • Script to benchmark the partitioning (PR)
  • Several minor fix (PR, PR, PR)

0.1.4

  • Unify the IO: One constructor to rule them all! (PR)
  • Octree and Geometry Objects Bug Fixes (PR)

0.1.3

  • Add KNN routines (KNN, KNN, KNN)
  • Unify API to load data (Point3DRDD, SphereRDD)
  • Speed-up cross-match methods by using native Scala methods (Scala)
  • Add a new website + spark3D belongs to AstroLab Software (website)
  • Update tutorials (tuto.
  • Few fixes here and there...

0.1.1

  • Add scripts to generate test data (PR)
  • Added Octree Partitioned RDD support (PR)
  • RDD[Sphere] added (SphereRDD) (PR)
  • Code refactoring: Replace Sphere by ShellEnvelope, Shape3D heritage, and move everything under geometryObjects (PR)
  • Add very simple Kryo registration for spark3D classes. Probably need more (PR, PR).
  • Few fixes here and there...

<= 0.1.0

Repo on fire! See from this PR and earlier commits. In short, what was available at that time (including bugs!):

  • Read and format data from external data sets. Coordinates can be spherical or cartesian.
    • Currently available: FITS and CSV data format.
  • Instantiate 3D objects.
    • Currently available: Point, Sphere, Spherical shell, Box.
  • Create RDD[T] from a raw RDD whose T is a 3D object. The new RDD has the same partitioning as the raw RDD.
    • Currently available: RDD[Point]
  • Re-partition RDD[T].
    • Currently available: Onion grid, Octree.
  • Identification and join methods between two data sets.
    • Currently available: cross-match between two RDD.