Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Commit

Permalink
Usermanagement documentation (Nifi) + Tensorflow use case (#130)
Browse files Browse the repository at this point in the history
* group management
* Tensorflow use case.
* NiFi - LDAP Documentation

Co-authored-by: Amen Ayadi <[email protected]>
Co-authored-by: Alexandre Nuttinck <[email protected]>
Co-authored-by: Sebastien Dupont <[email protected]>
  • Loading branch information
4 people authored May 4, 2021
1 parent 1a72f18 commit b383776
Show file tree
Hide file tree
Showing 18 changed files with 227 additions and 17 deletions.
140 changes: 124 additions & 16 deletions doc/USERMANAGEMENT.md

Large diffs are not rendered by default.

Binary file added doc/images/installation/Team1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/Team2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/Teams.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/create.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/grey.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/sign-in.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/installation/users.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/images/logos/tensorflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ This section contains various usage examples for FADI:

* [basic example](/USERGUIDE.md) with batch ingestion
* [streaming ingestion](examples/kafka/README.md) with streaming ingestion with the help of the [Apache Kafka](https://kafka.apache.org) message broker
* [on-demand compute environments](examples/binderhub/README.md) with [BinderHub](https://binderhub.readthedocs.io/en/latest/)
* [on-demand compute environments](examples/binderhub/README.md) with [BinderHub](https://binderhub.readthedocs.io/en/latest/)
* [Tensorflow example](examples/tensorflow/README.md) for image classification
101 changes: 101 additions & 0 deletions examples/tensorflow/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Tensorflow simple use case

This is a simple [Tensorflow](https://www.tensorflow.org/) and [Jupyter](https://jupyter.readthedocs.io/en/latest/) use case with FADI for image classification using the [MNIST dataset](http://yann.lecun.com/exdb/mnist/).


Before starting, change the environment:
* Click on `Control panel`
* Click on `Stop my server`
* Finally, click on `Start server`, choose `tensorflow environment` and click on `Spawn`.

![Jupyter web interface](./images/Tensorflow.png)


* Import TensorFlow:

```python
from __future__ import absolute_import, division, print_function, unicode_literals


import tensorflow as tf
```
* Load and prepare the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset. Convert the samples from integers to floating-point numbers:

```python
mnist = tf.keras.datasets.mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
```
* Build the `tf.keras.Sequential` model by stacking layers. Choose an optimizer and loss function for training:

```python
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])
```
* For each example the model returns a vector of "[logits](https://developers.google.com/machine-learning/glossary#logits)" or "[log-odds](https://developers.google.com/machine-learning/glossary#log-odds)" scores, one for each class.:

```python
predictions = model(x_train[:1]).numpy()
predictions
```
![Jupyter web interface](./images/Tensorflowusecase.png)

* The `tf.nn.softmax` function converts these logits to "probabilities" for each class:

```python
tf.nn.softmax(predictions).numpy()
```
![Jupyter web interface](./images/tensor2.png)

* The `losses.SparseCategoricalCrossentropy` loss takes a vector of logits and a True index and returns a scalar loss for each example.

```python
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
```
* This loss is equal to the negative log probability of the the true class: It is zero if the model is sure of the correct class.

This untrained model gives probabilities close to random (1/10 for each class), so the initial loss should be close to -tf.log(1/10) ~= 2.3.

```python
loss_fn(y_train[:1], predictions).numpy()
```
![Jupyter web interface](./images/tensor3.png)

```python
model.compile(optimizer='adam',
loss=loss_fn,
metrics=['accuracy'])
```
* The `Model.fit` method adjusts the model parameters to minimize the loss:

```python
model.fit(x_train, y_train, epochs=5)
```

![Jupyter web interface](./images/tensor4.png)

* The `Model.evaluate` method checks the models performance, usually on a "[Validation-set](https://developers.google.com/machine-learning/glossary#validation-set)".

```python
model.evaluate(x_test, y_test, verbose=2)
```
![Jupyter web interface](./images/tensor5.png)

* The image classifier is now trained to ~98% accuracy on this dataset, If you want your model to return a probability, you can wrap the trained model, and attach the softmax to it:

```python
probability_model = tf.keras.Sequential([
model,
tf.keras.layers.Softmax()
])
```

```python
probability_model(x_test[:5])
```
![Jupyter web interface](./images/tensor6.png)
Binary file added examples/tensorflow/images/Tensorflow.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/Tensorflowusecase.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added examples/tensorflow/images/tensor6.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit b383776

Please sign in to comment.