Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: duplicate read proto #1741

Merged
merged 24 commits into from
Apr 18, 2024
Merged

fix: duplicate read proto #1741

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
38fabe3
Fix duplicate read by cache.
Shylock-Hg Apr 16, 2024
a84c6f3
Lint fix.
Shylock-Hg Apr 16, 2024
6f28835
Remove RefCell.
Shylock-Hg Apr 16, 2024
7bdfd60
Lint fix.
Shylock-Hg Apr 16, 2024
f0c7193
Add option to control cache.
Shylock-Hg Apr 17, 2024
136c805
Lint fix.
Shylock-Hg Apr 17, 2024
a9f24cb
Lint fix.
Shylock-Hg Apr 17, 2024
67eabde
Lint fix.
Shylock-Hg Apr 17, 2024
8efe9c4
Use std::sync::Mutex.
Shylock-Hg Apr 17, 2024
76d61dd
Clone ResourceReader directly.
Shylock-Hg Apr 17, 2024
0811348
Temp commit.
Shylock-Hg Apr 17, 2024
a1dff86
Fix test.
Shylock-Hg Apr 17, 2024
7a3f902
Lint fix.
Shylock-Hg Apr 17, 2024
2134bd6
Merge branch 'fix/duplicate-read-proto-2' into fix/duplicate-read-pro…
Shylock-Hg Apr 17, 2024
1db5a38
Add trait bound.
Shylock-Hg Apr 17, 2024
c011dd1
Lint fix.
Shylock-Hg Apr 17, 2024
187ab41
Rename.
Shylock-Hg Apr 17, 2024
202c544
Lint fix.
Shylock-Hg Apr 17, 2024
f465172
restructure readers
tusharmath Apr 18, 2024
852c0e9
lint fixes
tusharmath Apr 18, 2024
da660a7
rename types
tusharmath Apr 18, 2024
7ee194b
lint fixes
tusharmath Apr 18, 2024
3a2d9f5
Update src/resource_reader.rs
tusharmath Apr 18, 2024
af2d812
drop reader implementation
tusharmath Apr 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@ convert_case = "0.6.0"
rand = "0.8.5"
tailcall-macros = { path = "tailcall-macros" }
tonic-types = "0.11.0"
async-lock = "3.3.0"


[dev-dependencies]
Expand Down
7 changes: 5 additions & 2 deletions src/config/reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
use std::collections::HashMap;
use std::path::Path;
use std::sync::Arc;

use async_lock::Mutex;
use rustls_pemfile;
use rustls_pki_types::{
CertificateDer, PrivateKeyDer, PrivatePkcs1KeyDer, PrivatePkcs8KeyDer, PrivateSec1KeyDer,
Expand All @@ -24,10 +26,11 @@ pub struct ConfigReader {

impl ConfigReader {
pub fn init(runtime: TargetRuntime) -> Self {
let cache: Arc<Mutex<HashMap<String, String>>> = Default::default();
tusharmath marked this conversation as resolved.
Show resolved Hide resolved
Self {
runtime: runtime.clone(),
resource_reader: ResourceReader::init(runtime.clone()),
proto_reader: ProtoReader::init(runtime),
resource_reader: ResourceReader::init(runtime.clone(), cache.clone()),
proto_reader: ProtoReader::init(runtime, cache.clone()),
}
}

Expand Down
2 changes: 1 addition & 1 deletion src/generator/generator.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ pub struct Generator {
}
impl Generator {
pub fn init(runtime: TargetRuntime) -> Self {
Self { proto_reader: ProtoReader::init(runtime) }
Self { proto_reader: ProtoReader::init(runtime, Default::default()) }
Shylock-Hg marked this conversation as resolved.
Show resolved Hide resolved
}

pub async fn read_all<T: AsRef<str>>(
Expand Down
12 changes: 7 additions & 5 deletions src/proto_reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
use std::collections::{HashMap, VecDeque};
use std::sync::Arc;

use anyhow::Context;
use async_lock::Mutex;
use futures_util::future::join_all;
use prost_reflect::prost_types::{FileDescriptorProto, FileDescriptorSet};
use protox::file::{FileResolver, GoogleFileResolver};
Expand All @@ -18,8 +20,8 @@ pub struct ProtoMetadata {
}

impl ProtoReader {
pub fn init(runtime: TargetRuntime) -> Self {
Self { resource_reader: ResourceReader::init(runtime) }
pub fn init(runtime: TargetRuntime, cache: Arc<Mutex<HashMap<String, String>>>) -> Self {
Self { resource_reader: ResourceReader::init(runtime, cache) }
}

pub async fn read_all<T: AsRef<str>>(&self, paths: &[T]) -> anyhow::Result<Vec<ProtoMetadata>> {
Expand Down Expand Up @@ -104,7 +106,7 @@ mod test_proto_config {
#[tokio::test]
async fn test_resolve() {
// Skipping IO tests as they are covered in reader.rs
let reader = ProtoReader::init(crate::runtime::test::init(None));
let reader = ProtoReader::init(crate::runtime::test::init(None), Default::default());
reader
.read_proto("google/protobuf/empty.proto")
.await
Expand Down Expand Up @@ -133,7 +135,7 @@ mod test_proto_config {
let runtime = crate::runtime::test::init(None);
let file_rt = runtime.file.clone();

let reader = ProtoReader::init(runtime);
let reader = ProtoReader::init(runtime, Default::default());
let helper_map = reader
.resolve_descriptors(reader.read_proto(&test_file).await?)
.await?;
Expand Down Expand Up @@ -177,7 +179,7 @@ mod test_proto_config {
#[tokio::test]
async fn test_proto_no_pkg() -> Result<()> {
let runtime = crate::runtime::test::init(None);
let reader = ProtoReader::init(runtime);
let reader = ProtoReader::init(runtime, Default::default());
let mut proto_no_pkg = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
proto_no_pkg.push("src/grpc/tests/proto_no_pkg.graphql");
let config_module = reader.read(proto_no_pkg.to_str().unwrap()).await;
Expand Down
36 changes: 32 additions & 4 deletions src/resource_reader.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
use std::collections::HashMap;
use std::sync::Arc;

use async_lock::Mutex;
use futures_util::future::join_all;
use futures_util::TryFutureExt;
use url::Url;
Expand All @@ -13,14 +17,39 @@

pub struct ResourceReader {
runtime: TargetRuntime,
// Cache file content, path->content
cache: Arc<Mutex<HashMap<String, String>>>,
Shylock-Hg marked this conversation as resolved.
Show resolved Hide resolved
}

impl ResourceReader {
pub fn init(runtime: TargetRuntime) -> Self {
Self { runtime }
pub fn init(runtime: TargetRuntime, cache: Arc<Mutex<HashMap<String, String>>>) -> Self {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pub fn init(runtime: TargetRuntime, cache: Arc<Mutex<HashMap<String, String>>>) -> Self {
pub fn init(runtime: TargetRuntime, cache: bool) -> Self {

If cache is true, we should initialize it inside of init

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And there is another issue, the first reading is called by ConfigReader::resource_reader, the second is called by ConfigReader::proto_reader, they are different object. So we need to get the HashMap from outside.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using Option<HashMap>?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And there is another issue, the first reading is called by ConfigReader::resource_reader, the second is called by ConfigReader::proto_reader, they are different object. So we need to get the HashMap from outside.

The cache should not be shared. If the cache needs to be shared, we should do that by sharing the ResourceReader.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's just an additional unwrapping that makes the code difficult to read. You can create two resource readers

struct ResourceReader<A>(A);

impl<A> ResourceReader<A> {
  fn direct(runtime: TargetRuntime) -> ResourceReader<Direct>
  fn cached(runtime: TargetRuntime) -> ResourceReader<Cache>
}

// Reads the files directly without cache
Direct {runtime: TargetRuntime }

// Reads it and maintains a cache
pub Cache {
  reader: Direct,
  cache: Arc<Mutex<HashMap<String, FileRead>>
}

Can we do something like above? We will know at compile time, if the reader is caching or not.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static dispatch is contagious, so I use the option to control whether enable cache.

Copy link
Contributor

@tusharmath tusharmath Apr 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes the code a lot more modular, type-safe and easy to test. The current implementation has a lot of checks before reading, increasing the potential of bugs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried it, but got compile error about not found self.resource_reader.read_file method of ProtoReader. I'm not familiar with this, maybe I need a trait bound to help compiler?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I switch to static dispatch and add a reader trait.

Self { runtime, cache }
}
/// Reads a file from the filesystem or from an HTTP URL
pub async fn read_file<T: ToString>(&self, file: T) -> anyhow::Result<FileRead> {
// check cache
let file_path = file.to_string();
let content = self
.cache
.lock()
.await

Check warning on line 35 in src/resource_reader.rs

View check run for this annotation

Codecov / codecov/patch

src/resource_reader.rs#L35 

Added line #L35 was not covered by tests
.get(&file_path)
.map(|v| v.to_owned());
let content = if let Some(content) = content {
content
} else {
let content = self.do_read_file(file.to_string()).await?;
self.cache
.lock()
.await

Check warning on line 44 in src/resource_reader.rs

View check run for this annotation

Codecov / codecov/patch

src/resource_reader.rs#L44 

Added line #L44 was not covered by tests
.insert(file_path.to_owned(), content.clone());
content
};

Ok(FileRead { content, path: file_path })
}

async fn do_read_file<T: ToString>(&self, file: T) -> anyhow::Result<String> {
// Is an HTTP URL
let content = if let Ok(url) = Url::parse(&file.to_string()) {
if url.scheme().starts_with("http") {
Expand All @@ -41,8 +70,7 @@

self.runtime.file.read(&file.to_string()).await?
};

Ok(FileRead { content, path: file.to_string() })
Ok(content)
}

/// Reads all the files in parallel
Expand Down
Loading