Skip to content

Commit

Permalink
Merge pull request #58 from quantifyearth/pf341-data-visualisation
Browse files Browse the repository at this point in the history
Data visualisation
  • Loading branch information
patricoferris authored Jul 7, 2024
2 parents 34349dd + 5078a56 commit e539e94
Show file tree
Hide file tree
Showing 10 changed files with 579 additions and 21 deletions.
21 changes: 13 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ the [promoted output version](./specs/shark.out.md).
$ patdiff -ascii specs/shark.md specs/shark.out.md
------ specs/shark.md
++++++ specs/shark.out.md
@|-1,40 +1,42 ============================================================
@|-1,41 +1,43 ============================================================
|
|# Markdown Shark Support
|
Expand All @@ -72,8 +72,8 @@ $ patdiff -ascii specs/shark.md specs/shark.out.md
|## Shark Build
|
-|```shark-build:gdal-env
+|```shark-build:gdal-env:ec610a45b8d858c2eba37fd40dd1764890828557c1c43fa84ec88c7fcdc087c1
|((from osgeo/gdal:ubuntu-small-3.6.3)
+|```shark-build:gdal-env:4213afafe74bc720d4bb210f21c97d54a361a80a838c248f26dd7dc019a40ac2
|((from ghcr.io/osgeo/gdal:ubuntu-small-3.6.3@sha256:bfa7915a3ef942b4f6f61223ee57eadbb469d6fb4a5fbf562286d1473f15eaab)
| (run (shell "mkdir -p /data && echo 'Something for the log!'")))
|```
|
Expand All @@ -83,16 +83,20 @@ $ patdiff -ascii specs/shark.md specs/shark.out.md
|## Shark Run
|
-|```shark-run:gdal-env
+|```shark-run:gdal-env:1dd3d7fdb8f1f485dd5aa0d5f383209a60aca98e67552d03a54c99be8b610eca
|$ gdalinfo --version > /data/gdal.version
-|$ gdalinfo --version > /data/gdal.version
-|$ curl -s https://france-geojson.gregoiredavid.fr/repo/regions/occitanie/region-occitanie.geojson > /data/region-occitanie.geojson
+|```shark-run:gdal-env:8113359387d02c4e29df7013f0cd2d699c4f1303d99fe065204b579baf0dd509
+|gdalinfo --version > /data/gdal.version
+|curl -s https://france-geojson.gregoiredavid.fr/repo/regions/occitanie/region-occitanie.geojson > /data/region-occitanie.geojson
|```
|
|Shark keeps track of inputs and outputs. In the next code block, Shark knows to wire
|up `/data/gdal.version` into the container.
|
-|```shark-run:gdal-env
+|```shark-run:gdal-env:e02469d800253ccf95e53b583e4a91465375a4e41479a67408331ecdeedb713e
|$ cat /data/gdal.version
-|$ cat /data/gdal.version
+|```shark-run:gdal-env:101cf72cd986ca89f23d87c6af4c86908385fd977ebdd4f010ebf6ae8d0b04c6
+|cat /shark/8113359387d02c4e29df7013f0cd2d699c4f1303d99fe065204b579baf0dd509/gdal.version
+|GDAL 3.6.3, released 2023/03/07
+|
|```
Expand All @@ -104,7 +108,8 @@ $ patdiff -ascii specs/shark.md specs/shark.out.md
|conventions to export data blobs.
|
|```shark-publish
|/data/gdal.version
-|/data/gdal.version
+|/obuilder-zfs/result/8113359387d02c4e29df7013f0cd2d699c4f1303d99fe065204b579baf0dd509/.zfs/snapshot/snap/rootfs/data/gdal.version
|```
[1]
```
Expand Down
2 changes: 2 additions & 0 deletions shark.opam
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ depends: [
"htmlit"
"routes"
"code-mirror"
"magic-mime"
"csv"

# Container-image deps / OBuilder
"mirage-crypto-rng"
Expand Down
3 changes: 2 additions & 1 deletion specs/shark.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ built and can be referenced as the context for future `shark-run` blocks.
## Shark Build

```shark-build:gdal-env
((from osgeo/gdal:ubuntu-small-3.6.3)
((from ghcr.io/osgeo/gdal:ubuntu-small-3.6.3@sha256:bfa7915a3ef942b4f6f61223ee57eadbb469d6fb4a5fbf562286d1473f15eaab)
(run (shell "mkdir -p /data && echo 'Something for the log!'")))
```

Expand All @@ -20,6 +20,7 @@ using that environment.

```shark-run:gdal-env
$ gdalinfo --version > /data/gdal.version
$ curl -s https://france-geojson.gregoiredavid.fr/repo/regions/occitanie/region-occitanie.geojson > /data/region-occitanie.geojson
```

Shark keeps track of inputs and outputs. In the next code block, Shark knows to wire
Expand Down
15 changes: 8 additions & 7 deletions specs/shark.out.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ built and can be referenced as the context for future `shark-run` blocks.

## Shark Build

```shark-build:gdal-env:ec610a45b8d858c2eba37fd40dd1764890828557c1c43fa84ec88c7fcdc087c1
((from osgeo/gdal:ubuntu-small-3.6.3)
```shark-build:gdal-env:4213afafe74bc720d4bb210f21c97d54a361a80a838c248f26dd7dc019a40ac2
((from ghcr.io/osgeo/gdal:ubuntu-small-3.6.3@sha256:bfa7915a3ef942b4f6f61223ee57eadbb469d6fb4a5fbf562286d1473f15eaab)
(run (shell "mkdir -p /data && echo 'Something for the log!'")))
```

Expand All @@ -18,15 +18,16 @@ using that environment.

## Shark Run

```shark-run:gdal-env:1dd3d7fdb8f1f485dd5aa0d5f383209a60aca98e67552d03a54c99be8b610eca
$ gdalinfo --version > /data/gdal.version
```shark-run:gdal-env:8113359387d02c4e29df7013f0cd2d699c4f1303d99fe065204b579baf0dd509
gdalinfo --version > /data/gdal.version
curl -s https://france-geojson.gregoiredavid.fr/repo/regions/occitanie/region-occitanie.geojson > /data/region-occitanie.geojson
```

Shark keeps track of inputs and outputs. In the next code block, Shark knows to wire
up `/data/gdal.version` into the container.

```shark-run:gdal-env:e02469d800253ccf95e53b583e4a91465375a4e41479a67408331ecdeedb713e
$ cat /data/gdal.version
```shark-run:gdal-env:101cf72cd986ca89f23d87c6af4c86908385fd977ebdd4f010ebf6ae8d0b04c6
cat /shark/8113359387d02c4e29df7013f0cd2d699c4f1303d99fe065204b579baf0dd509/gdal.version
GDAL 3.6.3, released 2023/03/07
```
Expand All @@ -38,5 +39,5 @@ this will publish to a `_shark` directory in the current working directory. Use
conventions to export data blobs.

```shark-publish
/data/gdal.version
/obuilder-zfs/result/8113359387d02c4e29df7013f0cd2d699c4f1303d99fe065204b579baf0dd509/.zfs/snapshot/snap/rootfs/data/gdal.version
```
2 changes: 1 addition & 1 deletion src/lib/md.ml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ let process_build_block ?(src_dir = ".") ?hb
in
Ast.Hyperblock.update_hash hb id;
let new_code_block =
let info_string = Block.to_info_string block in
let info_string = Block.to_info_string block_with_hash in
Cmarkit.Block.Code_block.make
~info_string:(info_string, Cmarkit.Meta.none)
(Cmarkit.Block.Code_block.code code_block)
Expand Down
97 changes: 97 additions & 0 deletions src/lib/server/build.ml
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
let ( / ) = Filename.concat

let find_paths pred paths =
let rec loop acc = function
| `Dir (_, more) -> List.fold_left loop acc more
| `File (path, _) -> if pred path then path :: acc else acc
| _ -> acc
in
loop [] paths

(* Rendering Builds *)
let render (Obuilder.Store_spec.Store ((module Store), store)) id =
let result, inputs =
Lwt_eio.run_lwt (fun () ->
let open Lwt.Syntax in
let* result = Store.result store id in
let+ inputs = Store.get_meta store id ":obuilder-run-input" in
( result,
match inputs with
| Some inputs ->
Some
(Obuilder.S.run_input_of_sexp (Sexplib.Sexp.of_string inputs))
| None -> None ))
in
match result with
| None ->
Cohttp_eio.Server.respond_string ~status:`OK ~body:("No result for " ^ id)
()
| Some path ->
let manifest, geojsons, jsons, images, tabular =
let src_dir = Fpath.(v path / "rootfs") |> Fpath.to_string in
match Obuilder.Manifest.generate ~exclude:[] ~src_dir "data" with
| Error (`Msg m) -> (m, [], [], [], [])
| Ok src_manifest ->
let geojsons =
find_paths
(fun p -> Filename.extension p = ".geojson")
src_manifest
in
let jsons =
find_paths (fun p -> Filename.extension p = ".json") src_manifest
in
let geojsons =
List.map (fun g -> Fmt.str "/file/%s/%s" id g) geojsons
in
let jsons =
List.map
(fun v ->
In_channel.with_open_bin (Filename.concat src_dir v)
@@ In_channel.input_all)
jsons
in
let images =
find_paths
(fun p ->
Filename.extension p = ".png"
|| Filename.extension p = ".jpeg")
src_manifest
in
let images =
List.map Uri.pct_encode images
|> List.map (fun v ->
"." / id
/ (Uri.with_query (Uri.of_string "serve")
[ ("file", [ v ]) ]
|> Uri.to_string))
in
let table =
let csvs =
find_paths (fun p -> Filename.extension p = ".csv") src_manifest
in
List.fold_left
(fun acc data ->
( In_channel.with_open_bin (Filename.concat src_dir data)
@@ fun ic -> Csv.load_in ic )
:: acc)
[] csvs
in
( Obuilder.Manifest.sexp_of_t src_manifest
|> Sexplib.Sexp.to_string_hum,
geojsons,
jsons,
images,
table )
in
let page =
Pages.build ~geojsons ~jsons ~images ~tabular ~manifest ~title:"Build"
~id ?inputs ()
in
let body =
Cohttp_eio.Body.of_string (Htmlit.El.to_string ~doctype:true page)
in
let headers =
(* Otherwise, an nginx reverse proxy will wait for the whole log before sending anything. *)
Cohttp.Header.init_with "X-Accel-Buffering" "no"
in
Cohttp_eio.Server.respond ~status:`OK ~headers ~body ()
2 changes: 1 addition & 1 deletion src/lib/server/dune
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@
(library
(name shark_server)
(public_name shark.server)
(libraries shark cohttp-eio htmlit routes))
(libraries shark cohttp-eio htmlit routes csv magic-mime))
Loading

0 comments on commit e539e94

Please sign in to comment.