Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove all init_logger part to use a common way for all crates. #3246

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Tocard
Copy link

@Tocard Tocard commented Nov 20, 2024

logging factorisation

To add json logging and avoid to add each logger into each crate, I'm wondering if we can use logger as libs instead.

json logging

The flag from the cli will provide json log if set or regular log to true.

That's the first ever contribution on rust, sorry in advance 👍

Code contributor checklist:

@Tocard Tocard changed the title Draft remove all init_logger part to use a common way for all crates. [Draft][WIP] remove all init_logger part to use a common way for all crates. Nov 20, 2024
@Tocard Tocard force-pushed the evol/refacto_logging_add_json branch from 82e5913 to 03e73c5 Compare November 20, 2024 22:04
@Tocard Tocard marked this pull request as draft November 20, 2024 22:19
@Tocard Tocard changed the title [Draft][WIP] remove all init_logger part to use a common way for all crates. remove all init_logger part to use a common way for all crates. Nov 20, 2024
@Tocard Tocard force-pushed the evol/refacto_logging_add_json branch from be53c30 to 11c3268 Compare November 20, 2024 22:20
Cargo.toml Outdated Show resolved Hide resolved
Copy link
Contributor

@teor2345 teor2345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR! Overall, I like the idea of refactoring our logging into a single crate, so it is consistent between binaries.

The logging crate refactor could go in a separate small PR, then adding JSON logging on top of that will be easier to review.

Adding a new crate also makes us think about what else we want to put in there. In this case, we might want a crate for all the things we do when we start a binary. But there’s no need to rename it yet, I want to check with the rest of the team first.

I don’t have a strong opinion about JSON logging, what are the use cases you’re trying to support?

crates/subspace-node/src/commands/run.rs Outdated Show resolved Hide resolved
crates/subspace-node/src/commands/run/consensus.rs Outdated Show resolved Hide resolved
crates/subspace-node/src/commands/wipe.rs Show resolved Hide resolved
crates/subspace-node/Cargo.toml Outdated Show resolved Hide resolved
Cargo.toml Outdated Show resolved Hide resolved
shared/subspace-logging/src/lib.rs Outdated Show resolved Hide resolved
@teor2345
Copy link
Contributor

Seems like our Windows CI has a bug which is impacting all our PRs. Sorry about that!

We use rustfmt and clippy to check our code style, do you need help setting them up?
Running cargo fmt --all should automatically fix formatting, and cargo clippy will give you a list of code style warnings to fix.

@Tocard
Copy link
Author

Tocard commented Nov 21, 2024

Thanks for this PR! Overall, I like the idea of refactoring our logging into a single crate, so it is consistent between binaries.

The logging crate refactor could go in a separate small PR, then adding JSON logging on top of that will be easier to review.

Adding a new crate also makes us think about what else we want to put in there. In this case, we might want a crate for all the things we do when we start a binary. But there’s no need to rename it yet, I want to check with the rest of the team first.

I don’t have a strong opinion about JSON logging, what are the use cases you’re trying to support?

I can split it into both Pr for sure.

As said into the description of the pull request, that's my first contribution into a rust project. I have no idea about architecture and layout suitable for the language haha.

We are many user into the project who do use logs to gather information and data for a lot of purpose. Json formatted logs are always better for this.

On my case

Logs <= Filebeat => Elasticsearch <=> Kibana / grafana to display stuff about my cluster health,

@Tocard
Copy link
Author

Tocard commented Nov 21, 2024

Seems like our Windows CI has a bug which is impacting all our PRs. Sorry about that!

We use rustfmt and clippy to check our code style, do you need help setting them up? Running cargo fmt --all should automatically fix formatting, and cargo clippy will give you a list of code style warnings to fix.
I tried to run it and it sayed all goods, Pretty obvious that was not the case haha.

I endup this pull-request later in night, will retry this evening

@Tocard
Copy link
Author

Tocard commented Nov 21, 2024

Seems like our Windows CI has a bug which is impacting all our PRs. Sorry about that!

We use rustfmt and clippy to check our code style, do you need help setting them up? Running cargo fmt --all should automatically fix formatting, and cargo clippy will give you a list of code style warnings to fix.

seems ok :) thanks

@Tocard Tocard force-pushed the evol/refacto_logging_add_json branch 4 times, most recently from 884d798 to 7b27f1a Compare November 21, 2024 18:00
@Tocard Tocard requested a review from teor2345 November 21, 2024 19:03
@Tocard
Copy link
Author

Tocard commented Nov 21, 2024

removed all json part for now

Copy link
Member

@nazar-pc nazar-pc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution!

Conceptually this makes a lot of sense to me and I think it is a good idea to do the refactoring first to extract common logic, then adding CLI option will be much easier.

The dependency only needs to be added to crates where it is used and remove tracing-subscriber from direct dependencies where you use subspace-logging since it will no longer be necessary.

shared/subspace-logging/src/lib.rs Outdated Show resolved Hide resolved
shared/subspace-logging/Cargo.toml Outdated Show resolved Hide resolved
shared/subspace-logging/Cargo.toml Outdated Show resolved Hide resolved
@Tocard Tocard force-pushed the evol/refacto_logging_add_json branch 2 times, most recently from 97a25d1 to 18dbe7a Compare November 21, 2024 20:33
Copy link
Contributor

@teor2345 teor2345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After removing the unused dependencies, we should be good to merge this

crates/subspace-node/Cargo.toml Outdated Show resolved Hide resolved
shared/subspace-logging/Cargo.toml Outdated Show resolved Hide resolved
crates/pallet-subspace/Cargo.toml Outdated Show resolved Hide resolved
@Tocard
Copy link
Author

Tocard commented Nov 22, 2024

Do there is a way to bild the whole project & binary to ensure all deps are fine ?

@Tocard Tocard force-pushed the evol/refacto_logging_add_json branch from 18dbe7a to 7bb1fa2 Compare November 22, 2024 12:31
@@ -97,5 +97,4 @@ binary = [
"dep:mimalloc",
"dep:subspace-metrics",
"dep:supports-color",
"dep:tracing-subscriber",
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what abot this one ? Shall we add dep::subspace-logging ?

if so why this one is needed only here and not to binary node ??

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is the feature for using the farmer crate as a binary, rather than a library. We only need to init logging when running the farmer binary, because libraries leave logging to the host application.

if so why this one is needed only here and not to binary node ??

The farmer crate contains a binary and library. The binary for the node is in subspace-node, but the subspace-service crate (and some other crates) are the libraries.

@Tocard Tocard marked this pull request as ready for review November 22, 2024 15:03
Copy link
Contributor

@teor2345 teor2345 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do there is a way to bild the whole project & binary to ensure all deps are fine ?

cargo clippy will find missing dependencies and unused imports.

Finding unused dependencies is harder. You could try cargo-udeps, but sometimes it will tell you that a dependency is missing, when it is actually used in an unusual way:
http://stackoverflow.com/questions/72082550/ddg#72082679

@@ -97,5 +97,4 @@ binary = [
"dep:mimalloc",
"dep:subspace-metrics",
"dep:supports-color",
"dep:tracing-subscriber",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is the feature for using the farmer crate as a binary, rather than a library. We only need to init logging when running the farmer binary, because libraries leave logging to the host application.

if so why this one is needed only here and not to binary node ??

The farmer crate contains a binary and library. The binary for the node is in subspace-node, but the subspace-service crate (and some other crates) are the libraries.

@@ -69,7 +70,6 @@ thread-priority = "1.1.0"
tokio = { version = "1.40.0", features = ["macros", "parking_lot", "rt-multi-thread", "signal", "sync", "time"] }
tokio-stream = { version = "0.1.16", features = ["sync"] }
tracing = "0.1.40"
tracing-subscriber = { version = "0.3.18", features = ["env-filter"], optional = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everywhere you have removed tracing-subscriber, please also remove supports-color.
(Sorry I didn’t catch this earlier!)

unsigned-varint = { version = "0.8.0", features = ["futures", "asynchronous_codec"] }
void = "1.0.2"
subspace-logging = { version = "0.0.1", path = "../../shared/subspace-logging" }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: we usually put crates in alphabetical order

@@ -55,6 +55,7 @@ subspace-erasure-coding = { version = "0.1.0", path = "../subspace-erasure-codin
subspace-farmer-components = { version = "0.1.0", path = "../subspace-farmer-components" }
subspace-core-primitives = { version = "0.1.0", path = "../subspace-core-primitives" }
subspace-kzg = { version = "0.1.0", path = "../../shared/subspace-kzg" }
subspace-logging = { version = "0.0.1", path = "../../shared/subspace-logging" }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The “deps” line changes this from an optional dependency into a required one for the binary only.

Suggested change
subspace-logging = { version = "0.0.1", path = "../../shared/subspace-logging" }
subspace-logging = { version = "0.0.1", path = "../../shared/subspace-logging", optional = true }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants