Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Miniscope-DAQ-V4 and CaImAn trigger #13

Merged
merged 73 commits into from
Apr 29, 2022
Merged
Show file tree
Hide file tree
Changes from 67 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
fa90040
remove 2p terms and adapt with inscopix miniscope
shenshan May 15, 2021
7c046a9
remove planes; preliminary separation of motion correction and segmen…
shenshan May 17, 2021
e53376f
Fix typos
shenshan May 17, 2021
79caca6
Update element_miniscope/miniscope.py
shenshan May 17, 2021
02792d9
Change a few table definitions and remove scan
shenshan May 18, 2021
12fff78
Merge branch 'dev' of https://github.com/shenshan/element-miniscope i…
shenshan May 18, 2021
6712f79
Delete CaImAn loader
kabilar May 19, 2021
f9ec391
Update element_miniscope/miniscope.py
shenshan May 21, 2021
59137e2
Update element_miniscope/miniscope.py
shenshan May 21, 2021
74e8f71
Update element_miniscope/miniscope.py
shenshan May 21, 2021
c0cdfe4
add back non-rigid motion correction
shenshan May 21, 2021
bbfb54a
Merge branch 'dev' of https://github.com/shenshan/element-miniscope i…
shenshan May 21, 2021
4faa0f9
Roi extraction -> segmentation
shenshan May 21, 2021
3a21168
remove scan module
shenshan Jul 23, 2021
fad1cf2
Merge branch 'dev' of https://github.com/shenshan/element-miniscope i…
kabilar Jul 27, 2021
0ca4c73
Merge pull request #7 from shenshan/dev
kabilar Sep 27, 2021
f6348ef
Update version
kabilar Sep 27, 2021
3373b0f
Move loader to `element_data_loader`
kabilar Sep 27, 2021
aa97bdc
Remove directory
kabilar Sep 27, 2021
ef9e423
Move loader to `element_data_loader`
kabilar Sep 27, 2021
c94b5a2
Clean up requirements
kabilar Sep 27, 2021
6c21706
Add timestamps & minor fixes
kabilar Sep 27, 2021
61460fa
Merge branch 'dev' of https://github.com/datajoint/element-miniscope …
kabilar Sep 27, 2021
4aeb097
Move populate routine from workflow to element
kabilar Oct 5, 2021
90833b2
Minor requirements update
kabilar Oct 5, 2021
a7a18de
Add support for Miniscope DAQ V4
kabilar Oct 18, 2021
542037d
Refactor to handle both V3 and V4 files
kabilar Oct 26, 2021
c9d2e5d
Update license
kabilar Mar 23, 2022
fc7111c
Update package name
kabilar Mar 23, 2022
4867cef
Add files for pypi release
kabilar Mar 23, 2022
31c2cac
Add issue templates
kabilar Mar 23, 2022
7704802
Merge branch 'main' into main
kabilar Mar 23, 2022
f725dd3
Remove MiniscopeAnalysis
kabilar Mar 23, 2022
cf239da
Merge branch 'main' of https://github.com/kabilar/element-miniscope i…
kabilar Mar 23, 2022
8e5fe8d
Move background to elements.datajoint.org
kabilar Apr 15, 2022
6b734ff
Add collapsible section
kabilar Apr 15, 2022
66b4603
Add citation section
kabilar Apr 16, 2022
5c738be
Add links to elements.datajoint.org
kabilar Apr 17, 2022
b25ca4c
Update format
kabilar Apr 18, 2022
1ebeb85
Revert separation of processing steps
kabilar Apr 24, 2022
0da6db5
Add processing and curation tables
kabilar Apr 24, 2022
c1523c4
Add directory functions
kabilar Apr 24, 2022
bb316d1
Update version
kabilar Apr 24, 2022
3a3bb3a
Update setup.py
kabilar Apr 24, 2022
c65e600
Fix RecordingInfo for Miniscope-DAQ-V4
kabilar Apr 24, 2022
d30ef31
Update comments
kabilar Apr 24, 2022
8726cdc
Add CaImAn triggering
kabilar Apr 24, 2022
9c86f09
Handle case of database conenction timeout
kabilar Apr 25, 2022
cf13d56
Remove z dimension and multiple fields
kabilar Apr 25, 2022
cde4458
Minor update to `Segmentation`. Add `MaskType`.
kabilar Apr 25, 2022
d8c5fb8
Minor `Fluorescence` update
kabilar Apr 25, 2022
3182a22
Remove processing_task_id attribute
kabilar Apr 26, 2022
d1ea911
Minor update to Activity table
kabilar Apr 26, 2022
fcc7925
Add comment
kabilar Apr 26, 2022
e99d578
Add changelog
kabilar Apr 26, 2022
9270039
Update README
kabilar Apr 26, 2022
27aecab
Add AnatomicalLocation table
kabilar Apr 26, 2022
acae116
Update diagram
kabilar Apr 26, 2022
6eeb4b0
Fix foreign key references
kabilar Apr 27, 2022
20e7500
Update element_miniscope/miniscope.py
kabilar Apr 27, 2022
e92e01b
Update to be nullable
kabilar Apr 27, 2022
cfbf9d1
Update element_miniscope/miniscope.py
kabilar Apr 27, 2022
c90614e
Update diagram
kabilar Apr 27, 2022
ac16e92
Merge branch 'main' of https://github.com/kabilar/element-miniscope
kabilar Apr 27, 2022
df4f15d
Update element_miniscope/miniscope.py
kabilar Apr 29, 2022
f0235e2
Update element_miniscope/miniscope.py
kabilar Apr 29, 2022
a496dd5
Update element_miniscope/miniscope.py
kabilar Apr 29, 2022
631a159
Update element_miniscope/miniscope.py
kabilar Apr 29, 2022
4e45816
Update element_miniscope/miniscope.py
kabilar Apr 29, 2022
651edc3
Update element_miniscope/miniscope.py
kabilar Apr 29, 2022
daea6c3
Update element_miniscope/miniscope.py
kabilar Apr 29, 2022
258e3f6
Update element_miniscope/miniscope.py
kabilar Apr 29, 2022
20e102d
Update element_miniscope/miniscope.py
kabilar Apr 29, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: 'bug'
assignees: ''

---

## Bug Report

### Description

A clear and concise description of what is the overall operation that is intended to be
performed that resulted in an error.

### Reproducibility
Include:
- OS (WIN | MACOS | Linux)
- DataJoint Element Version
- MySQL Version
- MySQL Deployment Strategy (local-native | local-docker | remote)
- Minimum number of steps to reliably reproduce the issue
- Complete error stack as a result of evaluating the above steps

### Expected Behavior
A clear and concise description of what you expected to happen.

### Screenshots
If applicable, add screenshots to help explain your problem.

### Additional Research and Context
Add any additional research or context that was conducted in creating this report.

For example:
- Related GitHub issues and PR's either within this repository or in other relevant
repositories.
- Specific links to specific lines or a focus within source code.
- Relevant summary of Maintainers development meetings, milestones, projects, etc.
5 changes: 5 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: DataJoint Contribution Guideline
url: https://docs.datajoint.org/python/community/02-Contribute.html
about: Please make sure to review the DataJoint Contribution Guidelines
57 changes: 57 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
name: Feature request
about: Suggest an idea for a new feature
title: ''
labels: 'enhancement'
assignees: ''

---

## Feature Request

### Problem

A clear and concise description how this idea has manifested and the context. Elaborate
on the need for this feature and/or what could be improved. Ex. I'm always frustrated
when [...]

### Requirements

A clear and concise description of the requirements to satisfy the new feature. Detail
what you expect from a successful implementation of the feature. Ex. When using this
feature, it should [...]

### Justification

Provide the key benefits in making this a supported feature. Ex. Adding support for this
feature would ensure [...]

### Alternative Considerations

Do you currently have a work-around for this? Provide any alternative solutions or
features you've considered.

### Related Errors
Add any errors as a direct result of not exposing this feature.

Please include steps to reproduce provided errors as follows:
- OS (WIN | MACOS | Linux)
- DataJoint Element Version
- MySQL Version
- MySQL Deployment Strategy (local-native | local-docker | remote)
- Minimum number of steps to reliably reproduce the issue
- Complete error stack as a result of evaluating the above steps

### Screenshots
If applicable, add screenshots to help explain your feature.

### Additional Research and Context
Add any additional research or context that was conducted in creating this feature request.

For example:
- Related GitHub issues and PR's either within this repository or in other relevant
repositories.
- Specific links to specific lines or a focus within source code.
- Relevant summary of Maintainers development meetings, milestones, projects, etc.
- Any additional supplemental web references or links that would further justify this
feature request.
147 changes: 147 additions & 0 deletions .github/workflows/development.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
name: Development
on:
pull_request:
push:
tags:
- '*.*.*'
jobs:
test-changelog:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Get changelog entry
id: changelog_reader
uses: guzman-raphael/changelog-reader-action@v5
with:
path: ./CHANGELOG.md
- name: Verify changelog parsing
env:
TAG_NAME: ${{steps.changelog_reader.outputs.version}}
RELEASE_NAME: Release ${{steps.changelog_reader.outputs.version}}
BODY: ${{steps.changelog_reader.outputs.changes}}
PRERELEASE: ${{steps.changelog_reader.outputs.status == 'prereleased'}}
DRAFT: ${{steps.changelog_reader.outputs.status == 'unreleased'}}
run: |
echo "TAG_NAME=${TAG_NAME}"
echo "RELEASE_NAME=${RELEASE_NAME}"
echo "BODY=${BODY}"
echo "PRERELEASE=${PRERELEASE}"
echo "DRAFT=${DRAFT}"
build:
needs: test-changelog
runs-on: ubuntu-latest
strategy:
matrix:
include:
- py_ver: 3.8
distro: alpine
image: djbase
env:
PY_VER: ${{matrix.py_ver}}
DISTRO: ${{matrix.distro}}
IMAGE: ${{matrix.image}}
DOCKER_CLIENT_TIMEOUT: "120"
COMPOSE_HTTP_TIMEOUT: "120"
steps:
- uses: actions/checkout@v2
- name: Compile image
run: |
export PKG_NAME=$(python3 -c "print([p for p in __import__('setuptools').find_packages() if '.' not in p][0])")
export PKG_VERSION=$(cat ${PKG_NAME}/version.py | awk -F\' '/__version__ = / {print $2}')
export HOST_UID=$(id -u)
docker-compose -f docker-compose-build.yaml up --exit-code-from element --build
IMAGE=$(docker images --filter "reference=datajoint/${PKG_NAME}*" \
--format "{{.Repository}}")
TAG=$(docker images --filter "reference=datajoint/${PKG_NAME}*" --format "{{.Tag}}")
docker save "${IMAGE}:${TAG}" | \
gzip > "image-${PKG_NAME}-${PKG_VERSION}-py${PY_VER}-${DISTRO}.tar.gz"
echo "PKG_NAME=${PKG_NAME}" >> $GITHUB_ENV
echo "PKG_VERSION=${PKG_VERSION}" >> $GITHUB_ENV
- name: Add image artifact
uses: actions/upload-artifact@v2
with:
name: image-${{env.PKG_NAME}}-${{env.PKG_VERSION}}-py${{matrix.py_ver}}-${{matrix.distro}}
path: "image-${{env.PKG_NAME}}-${{env.PKG_VERSION}}-py${{matrix.py_ver}}-\
${{matrix.distro}}.tar.gz"
retention-days: 1
- if: matrix.py_ver == '3.8' && matrix.distro == 'alpine'
name: Add pip artifacts
uses: actions/upload-artifact@v2
with:
name: pip-${{env.PKG_NAME}}-${{env.PKG_VERSION}}
path: dist
retention-days: 1
publish-release:
if: github.event_name == 'push'
needs: build
runs-on: ubuntu-latest
env:
TWINE_USERNAME: ${{secrets.twine_username}}
TWINE_PASSWORD: ${{secrets.twine_password}}
outputs:
release_upload_url: ${{steps.create_gh_release.outputs.upload_url}}
steps:
- uses: actions/checkout@v2
- name: Determine package version
run: |
PKG_NAME=$(python3 -c "print([p for p in __import__('setuptools').find_packages() if '.' not in p][0])")
SDIST_PKG_NAME=$(echo ${PKG_NAME} | sed 's|_|-|g')
PKG_VERSION=$(cat ${PKG_NAME}/version.py | awk -F\' '/__version__ = / {print $2}')
echo "PKG_NAME=${PKG_NAME}" >> $GITHUB_ENV
echo "PKG_VERSION=${PKG_VERSION}" >> $GITHUB_ENV
echo "SDIST_PKG_NAME=${SDIST_PKG_NAME}" >> $GITHUB_ENV
- name: Get changelog entry
id: changelog_reader
uses: guzman-raphael/changelog-reader-action@v5
with:
path: ./CHANGELOG.md
version: ${{env.PKG_VERSION}}
- name: Create GH release
id: create_gh_release
uses: actions/create-release@v1
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
with:
tag_name: ${{steps.changelog_reader.outputs.version}}
release_name: Release ${{steps.changelog_reader.outputs.version}}
body: ${{steps.changelog_reader.outputs.changes}}
prerelease: ${{steps.changelog_reader.outputs.status == 'prereleased'}}
draft: ${{steps.changelog_reader.outputs.status == 'unreleased'}}
- name: Fetch image artifact
uses: actions/download-artifact@v2
with:
name: image-${{env.PKG_NAME}}-${{env.PKG_VERSION}}-py3.8-alpine
- name: Fetch pip artifacts
uses: actions/download-artifact@v2
with:
name: pip-${{env.PKG_NAME}}-${{env.PKG_VERSION}}
path: dist
- name: Publish pip release
run: |
export HOST_UID=$(id -u)
docker load < "image-${{env.PKG_NAME}}-${PKG_VERSION}-py3.8-alpine.tar.gz"
docker-compose -f docker-compose-build.yaml run \
-e TWINE_USERNAME=${TWINE_USERNAME} -e TWINE_PASSWORD=${TWINE_PASSWORD} element \
sh -lc "pip install twine && python -m twine upload dist/*"
- name: Determine pip artifact paths
run: |
echo "PKG_WHEEL_PATH=$(ls dist/${PKG_NAME}-*.whl)" >> $GITHUB_ENV
echo "PKG_SDIST_PATH=$(ls dist/${SDIST_PKG_NAME}-*.tar.gz)" >> $GITHUB_ENV
- name: Upload pip wheel asset to release
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
with:
upload_url: ${{steps.create_gh_release.outputs.upload_url}}
asset_path: ${{env.PKG_WHEEL_PATH}}
asset_name: pip-${{env.PKG_NAME}}-${{env.PKG_VERSION}}.whl
asset_content_type: application/zip
- name: Upload pip sdist asset to release
uses: actions/upload-release-asset@v1
env:
GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
with:
upload_url: ${{steps.create_gh_release.outputs.upload_url}}
asset_path: ${{env.PKG_SDIST_PATH}}
asset_name: pip-${{env.SDIST_PKG_NAME}}-${{env.PKG_VERSION}}.tar.gz
asset_content_type: application/gzip
81 changes: 0 additions & 81 deletions Background.md

This file was deleted.

13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Changelog

Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.

## Unreleased - 2022-04-25
+ Add - Load data acquired with Miniscope-DAQ-V4
+ Add - Load data analyzed with CaImAn
+ Add - Trigger CaImAn analysis
+ Remove - Load data analyzed with MiniscopeAnalysis

## 0.1.1 - 2021-04-01
+ Add - Load data acquired with Miniscope-DAQ-V3
+ Add - Load data analyzed with MiniscopeAnalysis
15 changes: 15 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
ARG PY_VER
ARG DISTRO
ARG IMAGE
ARG PKG_NAME
ARG PKG_VERSION

FROM datajoint/${IMAGE}:py${PY_VER}-${DISTRO}
COPY --chown=anaconda:anaconda ./requirements.txt ./setup.py \
/main/
COPY --chown=anaconda:anaconda ./${PKG_NAME} /main/${PKG_NAME}
RUN \
cd /main && \
pip install . && \
rm -R /main/*
WORKDIR /main
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2021 DataJoint NEURO
Copyright (c) 2022 DataJoint

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
Loading