Skip to content

Commit

Permalink
Added some technical images to our docs and updated the mdbook
Browse files Browse the repository at this point in the history
  • Loading branch information
TimonPost committed Apr 4, 2019
1 parent 1363ffe commit 0852737
Show file tree
Hide file tree
Showing 15 changed files with 173 additions and 102 deletions.
8 changes: 8 additions & 0 deletions Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -95,5 +95,13 @@ pipeline {
}
}
}
stage('Publish book') {
when {
branch 'master'
}
steps{
echo 'Uploading book here'
}
}
}
}
8 changes: 7 additions & 1 deletion codecov.yml
Original file line number Diff line number Diff line change
@@ -1 +1,7 @@
comment: off
coverage:
precision: 2
round: down
range: "90...100"
ignore:
- "src/error"
- "Jeninsfile"
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ We need to try to avoid sending too much bandwidth in the first place, and then

There are a few methods we can implement to defeat congestion.
1. With [RTT](./rtt.md)
2. With [packet loss](./packet_loss.md).
2. With packet loss

3 changes: 1 addition & 2 deletions docs/md_book/src/congestion_avoidence/packet_loss.md
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
# Packet loss
To be done
# Whit Packet loss
9 changes: 6 additions & 3 deletions docs/md_book/src/congestion_avoidence/rtt.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Round Trip Time (RTT)
The time between you sending the packet and you receiving an acknowledgement from the other side is called RTT.
The time between you sending the packet and you receiving an acknowledgment from the other side is called RTT.
To avoid congestion we first need to find a way to calculate the `RTT` value of our connection so we can decide on top of that value if we have bad or good internet speeds.

_Smoothing factor_
Expand All @@ -10,7 +10,7 @@ _Allowed RTT value_

So now we have the smoothed RTT and or current RTT, GREAT! But RTT on its own is not bad. So there may be some max allowed RTT. We need to subtract that amount from our measured RTT multiplied by the smoothing factor.

The formule would look like the following:
The formula would look like the following:

```
// rtt_max_value is in ms
Expand All @@ -34,4 +34,7 @@ let new_rtt_value = (100 - 250) * 0.10.
As you see when or calculation is under 250ms we get a negative result, which is in this case positive.
When or calculation is above 250ms it will be positive, which is in this case negative.

So each time we receive an acknowledgement we can add our result, of the above formula, to the RTT time saved in the connection.
So each time we receive an acknowledgment we can add our result, of the above formula, to the RTT time saved in the connection.

## Interesting Reads
- [Wikipedia](https://en.wikipedia.org/wiki/Round-trip_delay_time)
23 changes: 11 additions & 12 deletions docs/md_book/src/fragmentation.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,24 @@
# Fragmentation
Fragmentation is dividing large packets into smaller fragments so that it can be sent over the network.

TCP will automatically divide packets into smaller parts if you send large amounts of data. But UDP doesn't support this fragmentation.
That is why we need to implement it or self.
TCP will automatically divide packets into smaller parts if you send large amounts of data. But UDP doesn't support fragmentation out-of-the-box.
Fortunately, laminar does.

How large may a packet be? We call this [MTU](https://en.wikipedia.org/wiki/Maximum_transmission_unit) which stands for maximum transmission unit.
Fragmentation will be applied to packets larger than the [MTU](https://en.wikipedia.org/wiki/Maximum_transmission_unit) with the following reliability types `Reliable Unordered`, `Reliable Ordered`, `Reliable Sequenced`.

What is this [MTU](https://en.wikipedia.org/wiki/Maximum_transmission_unit)? This stands for 'maximum transmission unit'.
On the Internet today (2016, IPv4) the real-world MTU is 1500 bytes.
When a packet is larger than 1500 bytes we need to split it up into different fragments.
Why 1500? That’s the default MTU for MacOS X and Windows.

So when we send a packet we need to follow this algorithm:
1. Check the size of the payload.
2. If the payload size is greater than the max allowed MTU divide the payload of the packet into x fragments.
3. Create a normal packet.
4. Send the fragments with the sequence number of the above parent's packet its sequence number.
5. When receiving the fragment store it in a buffer.
6. Once we have received all fragments combine the fragments and construct the payload for the user.

You should take note that each fragment will not be acknowledged with our implementation.
So if you would send 200.000 bytes (+- 133 fragments) the risk of one fragment being dropped will be huge.
If you really want to send large amounts of data over the line go for TCP instead, since that protocol is built for reliability and large data.

When sending small packets with the size of about 4000 bytes (4 fragments) this method will work fine. And won't probably cause any problems.
We are planning to support also sending larger packets with acknowledgments.re
We are planning to support also [sending larger packets](https://gafferongames.com/post/sending_large_blocks_of_data/) with acknowledgments.

## Interesting Reads
- [Gaffer about Fragmentation](https://gafferongames.com/post/packet_fragmentation_and_reassembly/)
- [Wikipedia](https://en.wikipedia.org/wiki/IP_fragmentation)
- [MTU](https://en.wikipedia.org/wiki/Maximum_transmission_unit)
63 changes: 21 additions & 42 deletions docs/md_book/src/packet_header.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,24 @@
# Packet Headers
In this topic we'll discuss the different headers we are pre-pending to the data sent via laminar.
In this topic we'll discuss the different headers we are pre-pending to the data sent via laminar.
We use different headers in different scenario's, we do this to reduce the packet size.

## Standard header
Will be included in each packet.
```rust
pub struct StandardHeader {
/// crc16 of the protocol version.
pub protocol_version: u16,
/// specifies the packet type.
pub packet_type_id: PacketTypeId,
/// specifies how this packet should be processed.
pub delivery_method: DeliveryMethod,
}
```
## Fragment header.
This is the header containing header fragment information prefix with tha standard header.
The first first fragment will also contain information the parent packet information.
```rust
pub struct FragmentHeader {
standard_header: StandardHeader,
// this is the sequence number to which the fragment belongs.
sequence: u16,
// this is the id from the fragment sho we know how to order the fragments on the other side.
id: u8,
// the number of fragments to which this fragment belongs.
num_fragments: u8,
// acked information of which will be included into the first fragment with id '1'.
packet_header: Option<AckedPacketHeader>,
}
```
Take a look over here: [image](LINK) for the complete design.

## Acked packet header.
This will be used for reliable packets.
```rust
pub struct AckedPacketHeader {
pub standard_header: StandardHeader,
/// this is the sequence number so that we can know where in the sequence of packages this packet belongs.
pub seq: u16,
// this is the last acknowledged sequence number.
ack_seq: u16,
// this is an bitfield of all last 32 acknowledged packages
ack_field: u32,
}
```
- `Standard header`

The first header is the `StandardHeader`, this is included for each packet.
It contains information like: protocol version, packet type, delivery and ordering guarantees.

- `AckedHeader`

This header will be included to the header if the packet is reliable.
It contains information for our acknowledgement system.

- `FragmentHeader`

This header will be included if the packet payload is bigger than the MTU and thus needs to be [fragmented](./fragmentation.md).

- `ArrangingHeader`

This header will be included if the packet needs to be arranged e.g ordered, sequenced.
It contains information like the stream it will be arranged on and an identifier for this packet.
6 changes: 1 addition & 5 deletions docs/md_book/src/reliability/basicis.md
Original file line number Diff line number Diff line change
@@ -1,5 +1 @@
# Acknowledgements

# Sequence Numbers

# Packet Loss
# Reliability
43 changes: 43 additions & 0 deletions docs/md_book/src/reliability/ordering.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
Laminar provides a way to arrange packets, over different streams.

The above sentence contains a lot of important information, let us zoom a little more on the above sentence.

## Ordering VS Sequencing
Let's define two concepts here:
_"Sequencing: this is the process of only caring about the newest items."_ [1](https://dictionary.cambridge.org/dictionary/english/sequencing)
_"Ordering: this is the process of putting something in a particular order."_ [2](https://dictionary.cambridge.org/dictionary/english/ordering)

- Sequencing: Only the newest items will be passed trough e.g. `1,3,2,5,4` which results in `1,3,5`.
- Ordering: All items are returned in order `1,3,2,5,4` which results in `1,2,3,4,5`.

### How ordering works.
Imagine we have this sequence: `1,5,4,2,3` and we want the user to eventually see: `1,2,3,4,5`.

## Arranging Streams
What are these 'arranging streams'?
You can see 'arranging streams' as something to arrange packets that have no relationship at all with one another.
You could either arrange packets in order or in sequence.

## Simple Example
Think of a highway where you have several lanes where cars are driving.
Because there are these lanes, cars can move on faster.
For example, the cargo drivers drive on the right and the high-speed cars on the left.
The cargo drivers have no influence on fast cars and vice versa.

## Real Example
If a game developer wants to send data to a client, it could happen that he wants to send data either ordered, unordered or sequenced.

'Data' could be the following:
1. Player movement, we want to order player movements because we don't want the player to glitch.
2. Bullet movement, we want to sequence bullet movement because we don't care about old positions of bullets.
3. Chat messages, we want to order chat messages because it is nice to see the text in the right order.

Player movement and chat messages are totally unrelated to each other and you absolutely do not want to interrupt the movement packets if a chat message is not sent.

It would be nice if we could order player movements and chat messages separately. Guess what! This is exactly what 'arranging streams' do.
A game developer can indicate which stream it likes to arrange the packets.
For example, the game developer can say: "Let me order all chat messages to 'stream 1' and sequence all motion packets on 'stream 2'.

## Interesting Reads
- [RakNet Ordering Streams](http://www.raknet.net/raknet/manual/sendingpackets.html)
- [LiteNetLib Implementation](https://github.com/RevenantX/LiteNetLib/issues/67)
88 changes: 52 additions & 36 deletions docs/md_book/src/reliability/reliability.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,75 +17,91 @@ _UDP_
- Unreliable.
- No guarantee for delivery.
- No guarantee for order.
- No way of getting dropped packet.
- No way of getting the dropped packet.
- Duplication possible.
- No [fragmentation](./../fragmentation.md).

So handy would be if you somehow could specify which features you want on top of UDP.
You could say for example I want the guarantee for my packets to arrive, however they don't need to be in order.
So it would be useful if we could somehow specify the features we want on top of UDP.
Like that you say: I want the guarantee for my packets to arrive, however they don't need to be in order.
Or, I don't care if my packet arrives but I do want to receive only new ones.

Laminar (will) provide(s) different kind of reliabilities as listed below:
Please check out [ordering documentation](ordering.md), it describes what ordering and sequencing is.

### Unreliable Unordered
Unreliable. Packets can be dropped, duplicated or arrive without order.
Laminar provides 5 different ways for you to send your data:

**Details**
| Reliability Type | Packet Drop | Packet Duplication | Packet Order | Packet Fragmentation |Packet Delivery|
| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: | :-------------:
| **Unreliable** | Yes | Yes | No | No | No
| **Unreliable Sequenced** | Yes | No | Sequenced | No | No
| **Reliable Unordered** | No | No | No | Yes | Yes
| **Reliable Ordered** | No | No | Ordered | Yes | Yes
| **Reliable Sequenced** | No | No | Sequenced | Yes | Yes


## Unreliable
Unreliable: Packets can be dropped, duplicated or arrive without order.

**Details**

| Packet Drop | Packet Duplication | Packet Order | Packet Fragmentation | Packet Delivery |
| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
| Yes | Yes | No | No | No |

Basically just bare UDP, free to be dropped, used for very unnecessary data, great for 'general' position
Basically just bare UDP. The packet may or may not be delivered.

### Unreliable Ordered
Unreliable. Packets can be dropped, duplicated or arrive with order.
// todo: add use cases

**Details**
## Unreliable Sequenced
Unreliable Sequenced: Packets can be dropped, but could not be duplicated and arrive in sequence.

*Details*

| Packet Drop | Packet Duplication | Packet Order | Packet Fragmentation | Packet Delivery |
| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
| Yes | Yes | Yes | No | No |
| Packet Drop | Packet Duplication | Packet Order | Packet Fragmentation | Packet Delivery |
| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
| Yes | Yes | Sequenced | No | No |

Basically just bare UDP, free to be dropped, used for very unnecessary data, great for 'general' position updates but packets will be ordered.
Basically just bare UDP, free to be dropped, but has some sequencing to it so that only the newest packets are kept.

// todo: add use cases

### Reliable Unordered
Reliable. All packets will be sent and received, but without order.
## Reliable Unordered
Reliable: All packets will be sent and received, but without order.

*Details*

| Packet Drop | Packet Duplication | Packet Order | Packet Fragmentation | Packet Delivery |
| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
| No | No | No | Yes | Yes |

Basically this is almost TCP like without ordering of packets.
Receive every packet and immediately give to application, order does not matter.
Basically, this is almost TCP without ordering of packets.

### Reliable Ordered
Reliable. All packets will be sent and received, with order.
// todo: add use cases
## Reliable Ordered
Reliable; All packets will be sent and received, with order.

*Details*

| Packet Drop | Packet Duplication | Packet Order | Packet Fragmentation | Packet Delivery |
| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
| No | No | Yes | Yes | Yes |
| No | No | Ordered | Yes | Yes |

Basically this is almost like TCP.

// todo: add use cases

Basically this is almost has all features TCP has.
Receive every packet (file downloading for example) in order (any missing keeps the later ones buffered.

### Sequenced
Unreliable. Packets can be dropped, but never duplicated and arrive in order.
## Reliable Sequenced
Reliable; All packets will be sent and received but arranged in sequence.
Which means that only the newest packets will be let through, older packets will be received but they won't get to the user.

*Details*

| Packet Drop | Packet Duplication | Packet Order | Packet Fragmentation | Packet Delivery |
| :-------------: | :-------------: | :-------------: | :-------------: | :-------------: |
| Yes | No | Yes | Yes | No |

Toss away any packets that are older than the most recent (like a position update, you don't care about older ones),
packets may be dropped, just the application may not receive older ones if a newer one came in first.

-----------------------------------------------------------------------------------
However all those options are listed above only a few will be supported for laminar version `0.1.0` like: UnreliableUnordered, ReliableUnordered, SequencedUnordered.
However for laminar version `0.2.0` we are planning to support: UnreliableOrdered, ReliableOrdered, SequencedOrdered also.

| No | No | Sequenced | Yes | Yes |

Basically this is almost TCP-like but then sequencing instead of ordering.

// todo: add use cases

## Interesting Reads
- [RakNet Reliability Types](http://www.jenkinssoftware.com/raknet/manual/reliabilitytypes.html)
22 changes: 22 additions & 0 deletions docs/technical/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
This folder contains some diagrams and images relteaded to laminar.

- `header_design.png`

This visualizes our packet headers.
- `incoming_packet_procssing_bpmn.png`

This is a BPMN-diagram that describes the incoming packet processing process.

- `ougoing_packet_procssing_bpmn.png`

This is a BPMN-diagram that describes the outgoing packet processing process.

- `ordering_sequencing_bpmn.png`

This is a BPMN-diagram showing the sequencing and ordering processes.

**Remarks**

Business Process Model and Notation (BPMN) is a graphical representation for specifying business processes in a business process model.
You could read more about that over [here](https://en.wikipedia.org/wiki/Business_Process_Model_and_Notation). It can also be applied to software processes.
I have to note here that there are some syntactic errors in those diagrams which are left to be resolved any time soon.
Binary file added docs/technical/hearder_design.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/technical/ordering_sequencing_bpmn.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 0852737

Please sign in to comment.