LSDB Demo at Rare Gems in Big Data

Demonstration prepared for the Rare Gems in Big Data 2024 meeting. This demo showcases working with HiPSCat tables via LSDB, and doing time domain analysis with TAPE.

See related documentation:

• HiPSCat (on GitHub) (on ReadTheDocs)
• LSDB (on GitHub) (on ReadTheDocs)
• TAPE (on GitHub) (on ReadTheDocs)

Demo time and place:

• 4.00 pm, Wednesday, May 22, in Kiva room

Relevant talks:

• 3.00 pm, Monday, May 20, Anastasios (Andy) Tzanidakis - presentation
• 11.15 am, Tuesday, May 21, Neven Caplar - presentation

Getting Started

You can follow along with this demo by creating your own local environment, or accessing the LINCC-hub (a shared cloud-hosted JupyterHub). We recommend using LINCC-hub because downloading the data stresses the Internet connection in the conference room, and the code will run much quicker and smoother if running via LINCC-hub.

Local installation

If installing in your own hardware, create a virtual environment then pip install relevant packages:

>> conda create --name lincc python=3.10
>> conda activate lincc
>> pip install lsdb lf-tape ipyaladin cesium aiohttp scikit-learn

Access LINCC Hub

1. You'll need an account on LINCC-hub. You can sign up by completing this form, and following the steps. Please complete this prior to attending the demo.
2. BEFORE STARTING YOUR SERVER, note that you should not use the default size! On the "Server Options" page select "Need more CPU or memory...?" and choose a "Large" server.
3. To get started, log into https://lsst.dirac.dev/. If this fails, reach out over slack on #lsdb_tape_tutorial and tag @nevencaplar.
4. After your server has started up (it will take 3 minutes at least!), clone this repo in LINCC-hub ("New" > "Terminal")

git clone https://github.com/lincc-frameworks/Rare_Gems_Demo

5. Open the notebooks in the "Rare Gems 2024" kernel. Running, especially the first cell, can take a minute, so feel free to run it before the demo starts if you want to follow along.
6. Work through the notebooks and have fun.
7. Shutdown each notebook after you're done to use less memory.

Notebooks

Notebook 1

Link to notebook

Overview:

• Learn how to (lazily) load catalogs
• Learn how to use those catalogs and perform crossmatching with existing LSDB catalogs
• Save the results

Notebook 2

Link to notebook

Overview:

In this notebook we will learn how to use the outputs from LSDB catalogs and use ensemble from TAPE to compute time-series features.

Notebook 3

Link to notebook

Overview:

• Learn how to use VizieR TAP query to access tables and store/handle them in LSDB
• Learn how to use those catalogs and perform crossmatching with existing LSDB catalogs
• Pass HipsCat LSDB catalogs to TAPE to perform time-series analysis and exploration

LINCC Tech-talks

Join on on Thursday, June 13, 10am Pacific, for the LINCC tech-talk to learn more about LSDB.

See more information (e.g., zoom link!) here.

Acknowledgements

This project is supported by Schmidt Sciences.