Carrot is an easy-to-use, real-time framework for building applications with multi-device AR capabilities. It works using WebSockets, Golang on the server-side, and a unique location tracking system based on iBeacons that we aptly named The Picnic Protocol. Using Carrot, multi-device AR apps can be created with high accuracy location tracking to provide rich and lifelike experiences. To see for yourself, check out Scribbles, a multiplayer drawing application made with Carrot. You can see a demo video here and the code here.
| 🗂 Table of Contents | |
|---|---|
| ✨ | Features |
| 📋 | To-Do |
| 🥗 | The Picnic Protocol |
| 🌎 | Sessions |
| ✉️ | Messages |
| 🎙 | Sending Messages to Carrot |
| 📨 | Receiving Messages from Carrot |
Here are some of what we think are Carrot's best features!
Carrot is completely separate from any particular renderer, leaving you free to render however you'd like! Carrot is also agnostic to any specific WebSocket implementation, leaving you in control of how you'd like to go about your networking. Not to mention, messages allow for any kind of information to get passed around between clients and taken advantage of by custom controllers on the server-side.
Carrot is real-time, lightning fast, and only getting faster thanks to some clever engineering on the backend. Carrot performs great even at around 30,000 requests per second, providing for a seamless multi-device AR experience.
Carrot is heavily customizable thanks to SessionDriver. Implement your own state management and authentication protocol while still being able to take advantage of the many other benefits Carrot provides.
Carrot allows you to take an existing AR app and make it multi-device without writing too much code, allowing you to focus on perfecting and shipping your AR experience! Initialize a session, render incoming messages, broadcast local rendering events and forget about it!
First off, we'd love as much input from the community as possible! Please don't hesitate to create issues for any problems encountered and PRs to get discussions kicked off about features you'd like to see Carrot implement. It's very early on in the framework's lifespan and we'd like to take this in whatever direction the community wants 🙂
Here are a few things we have in mind (in no particular order):
- Swift Package Manager support
- Custom
SessionDriverparity on the server-side - Error recovery commands
- Header fields in
Socketprotocol - Abstract away
PicnicProtocoliBeacon functionality into a protocol - More flexible route, endpoint & message paradigm (something like registering callbacks instead)
Brb, condensing an 11 page senior project report into a section of a Github README 🙃
Sessions are central to Carrot apps. A session is responsible for two things:
- Managing authentication via some underlying protocol.
- Providing a clean interface to the WebSocket used to relay messages to/from the Carrot server.
When it comes to sessions, the word protocol is referring to a set of rules that governs the communications between computers on a network, not the Swift keyword.
There are two types of sessions in Carrot: CarrotSession and CustomCarrotSession, and the difference between the two lies in the underlying protocol:
| Session | Driver |
|---|---|
CarrotSession |
PicnicProtocol |
CustomCarrotSession |
Custom protocol conforming to SessionDriver |
You may use start(stateDidChange: @escaping (Driver.State) -> Void) and end() in order to start/end sessions, respectively. Let's take a closer look at CarrotSession, which is the easiest way to get started with Carrot and the Picnic Protocol.
Under the hood, CarrotSession uses the PicnicProtocol class as its SessionDriver:
public final class CarrotSession<T: Codable>: CustomCarrotSession<PicnicProtocol, T> {
public init(
socket: Socket,
currentTransform: @escaping () -> matrix_float4x4?,
messageHandler: @escaping (MessageResult<T>) -> Void,
errorHandler: @escaping (PicnicProtocol.State?, Error) -> Void)
}The Socket protocol, declared in Socket.swift, allows you to provide Carrot with whatever underlying implemention of a WebSocket you wish. For an example using Facebook's SocketRocket, see CarrotSocket.swift.
Creating a CarrotSession within your UIViewController responsible for the ARSKView, for example, will look something like this:
carrotSession = CarrotSession(
socket: socket,
currentTransform: { [weak self] in
return self?.sceneView.session.currentFrame?.camera.transform
},
messageHandler: { result in
// handle receiving messages here
},
errorHandler: { _, error in
// handle errors here
})Opting for a CustomCarrotSession allows you to implement your own authentication protocol. This is designed for cases where maybe the Picnic Protocol is not a good fit for your multi-device AR experience. You're free to use whatever you'd like as long it conforms to the SessionDriver protocol, which we'll take a closer look at now.
At the time of writing, custom authentication protocols are not supported on the server-side. However, you can still use a CustomCarrotSession in order to implement the Picnic Protocol in a different way or to layer other logic on top of the Picnic Protocol.
A SessionDriver has two responsibilities:
- Managing state transitions via the
updateState(_:_:)method. - Managing state upon receiving data via the
didReceive(_:_:_:)method.
A SessionDriver is essentially a state-machine that responds to state changes and reacts to data that arrives via the WebSocket while in an unauthenticated state. Data that arrives while in an authenticated state gets passed to the session's messageHandler instead. Taking a look at DriverState should help clear this up.
SessionDriver "states" are represented by the DriverState protocol:
public protocol DriverState {
static var `default`: Self { get }
var token: SessionToken? { get }
var isAuthenticated: Bool { get }
}Conforming to DriverState correctly is very important, as it decides whether or not messages get passed to the SessionDriver or the session's consumer-facing messageHandler. This decision depends on the current state's isAuthenticated flag.
For an concrete example of conforming to SessionDriver, take a look at PicnicProtocol.swift. It codifies the Picnic Protocol rules, standards, and state management into an object conforming to SessionDriver and uses PicnicProtocolState and its DriverState.
Messages in Carrot are how information about events gets encoded and packaged for the server-side to broadcast to other clients in the same session. In Swift, they are represented by the Message<T: Codable> struct:
public struct Message<T: Codable> {
public var transform: matrix_float4x4?
public var object: T
}The transform property can be used to encode information about the position, orientation, and scale of objects pertaining to these encoded events, just like the corresponding property on ARAnchor in ARKit.
The generic object parameter allows you, the developer, to package any Codable information within a Message. This works nicely not only with Codable primitives like String, Bool, and Int but also with custom Codable classes, structs, and enums for example. Let's take a look at how this works in practice.
Using enum cases with associated values, you can describe different types of events within the same type, as long as the associated values are Codable. This allows you to represent the placement of different basic geometry nodes in an ARSKView with only one type (the custom conformance to Codable has been omitted for brevity):
enum Event: Codable {
case label(String)
case sphere(Int)
}
let message: Message<Event> = ...Little Bites of Cocoa #318 is a great tutorial on how to make an enum with associated values conform to Codable.
Let's say you want your app to communicate to other clients that you've placed an SKLabelNode somewhere in your ARSKView, for example. This would allow you to broadcast this information to other devices and build a multi-device AR experience instead of a just standard and boring one 😛
You'd be able to use the following struct to encode all the information about an SKLabelNode:
struct LabelEvent: Codable {
var text: String
}Now, you'd be able to construct a Message<LabelEvent> after you've created the ARAnchor that represents the position and orientation of this SKLabelNode that's been placed by the user:
let anchor = ARAnchor(transform: sceneView.session.currentFrame!.camera.transform)
let message = Message(transform: anchor.transform, object: LabelEvent(text: "🥕"))
// Send the message via my `CarrotSession`Upon receiving the message, another client would be able to decode it and have all the information necessary to render the SKLabelNode. We'll go over both the sending and receiving of messages in the following sections.
Sending messages happens via public func send(message: Message<Object>, to endpoint: Endpoint) throws. On top of the actual Message<Object> itself, this method requires an Endpoint. This Endpoint is how the server-side knows what controller to route the message to.
Using the LabelEvent we defined earlier, sending a message from the UIViewController responsible for the ARSKView would look something like this:
guard let currentTransform = sceneView.session.currentFrame?.camera.transform else { return }
let message = Message(
transform: currentTransform,
object: LabelEvent(text: "🥕"))
do {
try carrotSession.send(message: message, to: "draw")
} catch {
// handle the error here
}Receiving messages happens via the messageHandler closure of type (MessageResult<T>) -> Void that you provide in the initializer of a session. Messages get passed to this closure when the underlying SessionDriver's state is an authenticated one.
Thanks to first class functions in Swift, you might want to do something like this when you initialize your session:
carrotSession = CarrotSession(
socket: socket,
currentTransform: { [weak self] in
return self?.sceneView.session.currentFrame?.camera.transform
},
messageHandler: didReceive,
errorHandler: { _, error in
// handle errors here
})
func didReceive(messageResult result: MessageResult<LabelEvent>) {
switch result {
case let .success(message, endpoint):
// handle message here
case let .error(error):
// handle error here
}
}Here's an example of parsing a MessageResult and rendering it accordingly using the same function we define in the tip above:
var labels = [ARAnchor: LabelEvent]()
func didReceive(messageResult result: MessageResult<LabelEvent>) {
switch result {
case let .success(message, _):
// In this example, we ignore the endpoint since there's only one. We also know every `LabelEvent` is tied to a transform.
guard let transform = message.transform else { return }
let anchor = ARAnchor(transform: transform)
labels[anchor] = message.object
sceneView.add(anchor: anchor)
case let .error(error):
// handle error here
}
}Now, in the same UIViewController, we can implement ARSKViewDelegate's view(_ view: ARSKView, nodeFor anchor: ARAnchor):
func view(_ view: ARSKView, nodeFor anchor: ARAnchor) -> SKNode? {
guard let labelEvent = labels[anchor] else { return nil }
let labelNode = SKLabelNode(text: labelEvent.text)
// If we wanted to, we can give `LabelEvent` a way to codify the following alignment modes as well!
labelNode.horizontalAlignmentMode = .center
labelNode.verticalAlignmentMode = .center
return labelNode
}