The main repository for the 2024 hackathon with all the information for the hackathon and pointers to all repositories. Hackathon informaton pertaining to 2024 specifically will remain in here. The individual repositories are made to span multiple years. This years hackathon will continue to focus on ensuring the growth for each use case. This means the repositories created this year will be use again in subsequent years provided these use cases are repeated again. We need to ensure these solutions will lead to real world outcomes.
Bring developers from the Financial Services industry together to demonstrate how innovative Microsoft technologies can transform autism services by tackling real world use cases provided by families and creating lasting open-source projects for the community.
The date of the event is April 3rd and 4th. The event will be hybrid. The in-person venue will be in NYC at the EY Headquarter in 1 Manhattan West during World Autism Awareness Month (April).
An RBT (Registered Behavioral Technician) implements a session designed by the BCBA and looks to acomplish the goals and targets set by the BCBA. Usually, they carry an IPAD or an IoT device to capture their observations. But this has two challenges - it splits the focus of the RBT between focusing on the child and capturing the observations; and it is a very one-dimnesional approach to capture the observations. This use case attempts to solve these challenges by providing multi-sensory approach to data capture.
The session will comprise of:
- Multi-modal data capture
- A smartphone app used by RBT - similar to what some RBTs have today
- Video camera to record the session - this will focus on capturing facial expressions, body language, fluency of tasks, etc.
- Microphone - this will focus on capturing voice, tone, etc. to determine emotions
- Real-time identification of pre-designed triggers
- This module will specifically look for pre-designed triggers - emotional (frustration, accomplishment, etc.), or physical (completion of task, eye contact, etc.)
- The objective is to track progress along specific markers
- Post-session summarization
- Summarize and generate notes - free form based on observations across all modes
- Summarize progress on specific markers
- RBT, BCBA, parent, child
| Name | Role | Company |
|---|---|---|
| Rishi Bhatnagar | Use Case Lead | Morgan Stanley |
| Subject Matter Expert | RBT/BCBA | |
| Tech Lead | Microsoft | |
| Tech Lead | EY |
Project Florence -Vision Studio for Video/Image Analysis
A BCBA (Board Certified Behavior Analyst) designs a program specifically for each child based on individualized treatment plan. It will center around the activitie the child enjoys and responds to. The objective is to have specific tasks for the child to complete with a certain level of fluency. This needs to take into account previous assessments, observations, and progress markers.
- The co-pilot will be able to ingest past program summaries in multi-modal form - videos, notes (digitized and undigitized), voide recordings, etc
- Based on these, the co-pilot will place the child in a cohort of other children with similar assessments and markers
- Based on past successes of children in the cohort, the co-pilot will suggest a recalibrated treatment and markers (if needed) to the BCBA
- It can generate a progress summary at any given point of time
- RBT, BCBA, parent, child
| Name | Role | Company |
|---|---|---|
| Rishi Bhatnagar | Use Case Lead | Morgan Stanley |
| Subject Matter Expert | BCBA | |
| Tech Lead | Microsoft | |
| Tech Lead | EY |
Typically, a BCBA gets limited amount of time to onboard a new patient and this can be a very cumbersome task. It includes several types of information to intake:
- Patient demographics, medical history, past services, etc.
- Historical evaluations, IEPs, etc.
- Insurance, funding, billing information, etc.
- Previous and current treatments and their outcomes, progress reports, etc.
- Consent forms, contracts, letter of engagement, etc.
- The co-pilot will be able to ingest historical documents
- It will allow parents, BCBAs, and RBTs to actively collaborate across sessions - such as allowing parents to provide inputs of progress outside of sessions
- It will allow tracking documents and provide a platform for storing consents, authorizations, etc.
- It can generate a summarization of the new client for the BCBA to review
- BCBA, parent, child
| Name | Role | Company |
|---|---|---|
| Leo Junquera | Use Case Lead | EY |
| Leo Junquera | Subject Matter Expert | Parent |
| Tech Lead | Microsoft | |
| Tech Lead | EY |
Objective: Connect parents of children with ASD with helpful resources in the community - other parents, BCBAs, organizations, etc.
When parents move into a new neighborhood, they are not sure what local resources are available to help them or their child. They don't know which RBT, BCBA, or ABA therapist will be best suited for their child. Often times, parents don't even know what questions to ask or what benefits are available to them. This app attempts to start solving the problem.
This mobile app will help connect parents and patients with other helpful members of the community - other parents, BCBAs, organizations, etc. It will work in three main phases
- Intake
- Allow patients and parents to fill in their and thier child's profile
- Offer a helpful "onbording survey" that will gather basic information about their child
- Discover
- Use ChatGPT to allow parents and patients ask open ended questions and have a conversation
- Prompt the right next actions for the parents
- Provide helpful information from credible sources
- Connect
- Connect them with other parents in same situation
- Connect them with credible resources and organizations that offer help in the specific area they are looking for
- RBT, BCBA, parent, child, non-profit organization
| Name | Role | Company |
|---|---|---|
| Dhruv Bhatnagar | Use Case Lead | Student |
| Crystal | Subject Matter Expert | |
| Tech Lead | Microsoft | |
| Tech Lead | EY |
Theme 2: Ideas that are leverage innovative and exploratory technology and have a longer time to market
This use case will focus on the use of HoloLens for teaching job skills to people with Autism, particularly those aging out of the supports provided in the school environment. Unemployment for those with Autism is significant and the supports in the work environment are limited. The use case will use HoloLens AR to help with basic job skills by presenting visual cues on how to complete task such as stocking shelves or preparing food. Additonally, the use case will make use of Gen AI, to manage, evaluate, and provide feedback to the student on how well the skill is being performed. It can also be used to generated the slight variants and increasingly complex scenarios to practice and introduce gamification. This will also address the issue of scaability - moving away from 1:1 support to a 1:N as an a student becomes an adult.
The skills th application will focus on can include one or more of the following:
- Stacking shelves in a store – e.g., making sure all the labels are facing outwards, etc.
- Setting up a table in a restaurant – making sure all the cutlery is there an arranged properly
- Setting up a catering tray – e.g., making sure all items are placed properly
| Name | Role | Company |
|---|---|---|
| Use Case Lead | ||
| Subject Matter Expert | BCBA | |
| Tech Lead | Microsoft | |
| Tech Lead | EY |
The world is filled with challenging social situations. People facing new situations often struggle to find appropriate responses. For those on the autism spectrum, this is exacerbated by difficulty reading standard social cues. Having a safe and controlled environment to practice responding to common social situations, personality types, and social cues would allow users to face new situations with confidence. Since creating such an environment would be difficult to scale, this app applies augmented reality to provide a unique social practice environment in an application. Furthermore, since those with autism are often drawn to repetition, the app utilizes generative AI to ensure adequate variety across sessions. Generative AI will also be used to evaluate the user’s performance.
This augmented reality app will present users with a variety of interactive stories that include common social situations and personality types. Users progress through the story by meeting a variety of characters with the goal of successfully completing the story. Each story consists of the following components:
- Story board – Configuration of the story space and VR coordinates. This section is designed in conjunction with VR designers.
- Characters – Configuration of the main characters and their general personality types. The personality types are fed as prompts to generative AI to script the characters’ initial comments and subsequent responses to the user.
- Scoring – Analysis of the user’s handling of the situation is performed by feeding the entire conversation to generative AI for analysis of the user’s reactions. Advancement in the game is based on the score.
| Name | Role | Company |
|---|---|---|
| Stephen Goldbaum | Use Case Lead | Morgan Stanley |
| Subject Matter Expert | BCBA | |
| Tech Lead | Microsoft | |
| Tech Lead | EY |
We often get asked "how can I participate" and have therefore outlined a list of roles and their responsiblities.
Curate resources and organize sessions in order to equip participants with the knowledge needed to prepare for the hackathon.
- Build technical training guide with learning geared towards each use case
- Put together contextual training guide focused on autism - current state, challenges, and gaps
- Organize and distribute training schedule, including “ask me anything” and technical/contextual skilling sessions
Serve as the administrator of a use case. Ensure that both the tech leads and participants have what they need before and during the hackathon.
- Support planning efforts and help in aligning 2-3 tech leads to use case
- Organize prep sessions leading up to the hackathon and ensure participants are connected with tech leads
- Oversee activity and logistics during the two-day hackathon
- Work with Tech Lead to capture summary/results
Serve as technical point of contact for a use case, reviewing output from previous years and helping to unblock technical barriers against use case objectives.
- Provide recommendations and help to finalize use case
- Lead prep sessions for participants and be available for technical questions
- Manage sponsored subscriptions and set up for participants
- Lead and guide team(s) during the two-day hackathon
- Work with Use Case Lead to capture summary/results
Organize marketing material for the Hackathon and generate interest internally and externally.
- Build a folder of references including:
- Word and PowerPoint branded templates
- Brochure to be distributed to interested external organizations
- One-pager highlighting the event
Ensure that everything is running smoothly across technical planning details of the Hackathon.
- Partner with individuals building registration platform to oversee and manage registration, maintaining organized list
- Point of contact to navigate any external access issues
- Create Team that includes organize training content, use case details, and channels and assist with adding/managing members
- Responsible for day-of operations, overseeing schedule and making sure everyone knows where they are supposed to be
Serve as the point person for a given customer organization, ensuring they have what they need to prepare their developers for the hackathon.
- Identify customer “champion” lead who will be your primary liaison for the hackathon
- Work with Marketing Lead and Training Leads to get content and resources to share with your customer
- Keep your customer informed on a regular basis on registration timelines, training sessions, and other event logistics