Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AVRO-4087: Support .NET 9 #3231

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

AVRO-4087: Support .NET 9 #3231

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
94e4cee
Support .NET 9
Oct 30, 2024
073ab0c
Remove comment
Oct 30, 2024
5676b83
Install .NET 9 SDK for ARM64 build
Oct 30, 2024
0de9300
.NET 9 GA
Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/codeql-csharp-analysis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ jobs:
6.0.x
7.0.x
8.0.x
9.0.x

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/test-lang-csharp-ARM.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ jobs:
6.0.x
7.0.x
8.0.x
9.0.x

- uses: actions/cache@v4
with:
Expand Down Expand Up @@ -79,6 +80,7 @@ jobs:
6.0.x
7.0.x
8.0.x
9.0.x

- name: Cache Local Maven Repository
uses: actions/cache@v4
Expand Down Expand Up @@ -145,6 +147,7 @@ jobs:
bash ./dotnet-install.sh --channel "6.0" --install-dir "$HOME/.dotnet" # 6.0
bash ./dotnet-install.sh --channel "7.0" --install-dir "$HOME/.dotnet" # 7.0
bash ./dotnet-install.sh --channel "8.0" --install-dir "$HOME/.dotnet" # 8.0
bash ./dotnet-install.sh --channel "9.0" --install-dir "$HOME/.dotnet" # 9.0

- name: Build
run: |
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/test-lang-csharp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ jobs:
6.0.x
7.0.x
8.0.x
9.0.x

- uses: actions/cache@v4
with:
Expand Down Expand Up @@ -79,6 +80,7 @@ jobs:
6.0.x
7.0.x
8.0.x
9.0.x

- name: Cache Local Maven Repository
uses: actions/cache@v4
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/test-lang-java.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ jobs:
6.0.x
7.0.x
8.0.x
9.0.x

- name: 'Create Interop Data Directory'
working-directory: .
Expand Down
30 changes: 15 additions & 15 deletions lang/csharp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,25 +12,25 @@ Install-Package Apache.Avro

## Build & Test

1. Install [.NET SDK 8.0+](https://dotnet.microsoft.com/download/dotnet-core)
1. Install [.NET SDK 9.0+](https://dotnet.microsoft.com/download/dotnet-core)
2. `dotnet test`

## Project Target Frameworks

| Project | Published to nuget.org | Type | .NET Standard 2.0 | .NET Standard 2.1 | .NET 6.0 | .NET 7.0 | .NET 8.0 |
|:-------------------:|:--------------------------:|:----------:|:------------------:|:-----------------:|:---------:|:---------:|:---------:|
| Avro.main | Apache.Avro | Library | ✔️ | ✔️ | | | |
| Avro.File.Snappy | Apache.Avro.File.Snappy | Library | ✔️ | ✔️ | | | |
| Avro.File.BZip2 | Apache.Avro.File.BZip2 | Library | ✔️ | ✔️ | | | |
| Avro.File.XZ | Apache.Avro.File.XZ | Library | ✔️ | ✔️ | | | |
| Avro.File.Zstandard | Apache.Avro.File.Zstandard | Library | ✔️ | ✔️ | | | |
| Avro.codegen | Apache.Avro.Tools | Exe | | |✔️ |✔️ |✔️ |
| Avro.ipc | | Library | ✔️ | ✔️ | | | |
| Avro.ipc.test | | Unit Tests | | |✔️ |✔️ |✔️ |
| Avro.msbuild | | Library | ✔️ | ✔️ | | | |
| Avro.perf | | Exe | | |✔️ |✔️ |✔️ |
| Avro.test | | Unit Tests | | |✔️ |✔️ |✔️ |
| Avro.benchmark | | Exe | | |✔️ |✔️ |✔️ |
| Project | Published to nuget.org | Type | .NET Standard 2.0 | .NET Standard 2.1 | .NET 6.0 | .NET 7.0 | .NET 8.0 | NET 9.0 |
|:-------------------:|:--------------------------:|:----------:|:------------------:|:-----------------:|:---------:|:---------:|:---------:|:---------:|
| Avro.main | Apache.Avro | Library | ✔️ | ✔️ | | | | |
| Avro.File.Snappy | Apache.Avro.File.Snappy | Library | ✔️ | ✔️ | | | | |
| Avro.File.BZip2 | Apache.Avro.File.BZip2 | Library | ✔️ | ✔️ | | | | |
| Avro.File.XZ | Apache.Avro.File.XZ | Library | ✔️ | ✔️ | | | | |
| Avro.File.Zstandard | Apache.Avro.File.Zstandard | Library | ✔️ | ✔️ | | | | |
| Avro.codegen | Apache.Avro.Tools | Exe | | |✔️ |✔️ |✔️ |✔️ |
| Avro.ipc | | Library | ✔️ | ✔️ | | | | |
| Avro.ipc.test | | Unit Tests | | |✔️ |✔️ |✔️ |✔️ |
| Avro.msbuild | | Library | ✔️ | ✔️ | | | | |
| Avro.perf | | Exe | | |✔️ |✔️ |✔️ |✔️ |
| Avro.test | | Unit Tests | | |✔️ |✔️ |✔️ |✔️ |
| Avro.benchmark | | Exe | | |✔️ |✔️ |✔️ |✔️ |

## Dependency package version strategy

Expand Down
4 changes: 2 additions & 2 deletions lang/csharp/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ do

perf)
pushd ./src/apache/perf/
dotnet run --configuration Release --framework net8.0
dotnet run --configuration Release --framework net9.0
;;

dist)
Expand Down Expand Up @@ -77,7 +77,7 @@ do
;;

interop-data-generate)
dotnet run --project src/apache/test/Avro.test.csproj --framework net8.0 ../../share/test/schemas/interop.avsc ../../build/interop/data
dotnet run --project src/apache/test/Avro.test.csproj --framework net9.0 ../../share/test/schemas/interop.avsc ../../build/interop/data
;;

interop-data-test)
Expand Down
11 changes: 5 additions & 6 deletions lang/csharp/common.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@

<PropertyGroup Label="Target Frameworks">
<!-- Exe -->
<DefaultExeTargetFrameworks>net6.0;net7.0;net8.0</DefaultExeTargetFrameworks>
<DefaultExeTargetFrameworks>net6.0;net7.0;net8.0;net9.0</DefaultExeTargetFrameworks>
<!-- Library -->
<DefaultLibraryTargetFrameworks>netstandard2.0;netstandard2.1</DefaultLibraryTargetFrameworks>
<!-- Unit Tests -->
Expand Down Expand Up @@ -80,11 +80,10 @@
<EditorConfigFiles Condition="'$(IsTestProject)' == 'true'" Include="$(MSBuildThisFileDirectory)/CodeAnalysis.test.globalconfig" />
</ItemGroup>

<ItemGroup Condition="'$(RunAnalyzers)' != 'false'">
<PackageReference Include="Microsoft.CodeAnalysis.NetAnalyzers" Version="$(MicrosoftCodeAnalysisNetAnalyzersVersion)" PrivateAssets="all" />
<PackageReference Include="Microsoft.CodeAnalysis.CSharp.CodeStyle" Version="$(MicrosoftCodeAnalysisCSharpCodeStyleVersion)" PrivateAssets="all" />
<PackageReference Include="StyleCop.Analyzers" Version="$(StyleCopAnalyzersVersion)" PrivateAssets="all" />
</ItemGroup>
<PropertyGroup Condition="'$(RunAnalyzers)' != 'false'">
<EnableNETAnalyzers>true</EnableNETAnalyzers>
<AnalysisLevel>latest</AnalysisLevel>
</PropertyGroup>

<Target Name="PrintVersionInfo">
<Message Importance="high" Text="VersionPrefix: $(VersionPrefix)" />
Expand Down
3 changes: 1 addition & 2 deletions lang/csharp/src/apache/benchmark/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,7 @@ namespace Avro.Benchmark
{
public class Program
{
// dotnet run -c Release -f net8.0
// dotnet run -c Release -f net8.0 --runtimes net6.0 net7.0 net8.0
// dotnet run -c Release -f net9.0
public static void Main(string[] args)
{
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
Expand Down
21 changes: 9 additions & 12 deletions lang/csharp/versions.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@
!!! SHIPPED CLASS LIBRARIES SHOULD USE MINIMUMVERSIONs FOR SOME LIBRARIES. SEE BELOW !!!
-->
<PropertyGroup Label="Latest Package Versions">
<NewtonsoftJsonVersion>13.0.1</NewtonsoftJsonVersion>
<SystemCodeDomVersion>8.0.0</SystemCodeDomVersion>
<NewtonsoftJsonVersion>13.0.3</NewtonsoftJsonVersion>
<SystemCodeDomVersion>9.0.0</SystemCodeDomVersion>
<SystemReflectionVersion>4.3.0</SystemReflectionVersion>
<SystemReflectionEmitILGenerationVersion>4.7.0</SystemReflectionEmitILGenerationVersion>
<SystemReflectionEmitLightweightVersion>4.7.0</SystemReflectionEmitLightweightVersion>
Expand Down Expand Up @@ -58,19 +58,16 @@
Please sort the packages alphabetically
-->
<PropertyGroup Label="Build, Test, Code Analysis, Benchmark Package Versions">
<BenchmarkDotNetVersion>0.13.10</BenchmarkDotNetVersion>
<CoverletCollectorVersion>6.0.0</CoverletCollectorVersion>
<CoverletMSBuildVersion>6.0.0</CoverletMSBuildVersion>
<MicrosoftBuildFrameworkVersion>17.8.3</MicrosoftBuildFrameworkVersion>
<MicrosoftBuildUtilitiesCoreVersion>17.8.3</MicrosoftBuildUtilitiesCoreVersion>
<MicrosoftCodeAnalysisVersion>4.7.0</MicrosoftCodeAnalysisVersion>
<MicrosoftCodeAnalysisCSharpVersion>4.7.0</MicrosoftCodeAnalysisCSharpVersion>
<MicrosoftCodeAnalysisCSharpCodeStyleVersion>4.7.0</MicrosoftCodeAnalysisCSharpCodeStyleVersion>
<MicrosoftCodeAnalysisNetAnalyzersVersion>8.0.0</MicrosoftCodeAnalysisNetAnalyzersVersion>
<BenchmarkDotNetVersion>0.14.0</BenchmarkDotNetVersion>
<CoverletCollectorVersion>6.0.2</CoverletCollectorVersion>
<CoverletMSBuildVersion>6.0.2</CoverletMSBuildVersion>
<MicrosoftBuildFrameworkVersion>17.12.6</MicrosoftBuildFrameworkVersion>
<MicrosoftBuildUtilitiesCoreVersion>17.12.6</MicrosoftBuildUtilitiesCoreVersion>
Comment on lines +64 to +65
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These MSBuild version changes worry me. The properties are used in lang/csharp/src/apache/msbuild/Avro.msbuild.csproj so does the custom task assembly become impossible to load into an older version of MSBuild, e.g. in an older version of Visual Studio? There is a comment above the PropertyGroup that claims these are safe to upgrade, but I'm not sure it is true.

In contrast, the Microsoft.CodeAnalysis APIs are used in tests only, so those packages are safe to uograde.

Copy link
Contributor Author

@zcsizmadia zcsizmadia Oct 30, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MSBuilkd package is not distributed to nuget and that source was not maintained for a long time. Additionally IMO it should be retireed from the source code, because avrogen is the tool for code generation and MS recommends not using MSBuild task for more complex cases. Additionally to this AFAIK MSBuild tasks are backwards compatible with older (to a certain level) VS IDEs. Mostly they are netstandard2.0 libraries with standard interfaces and the MSBuild.Framework librariries do not come really into play with the executing VS IDE.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Avro.MSBuild is not being published in a NuGet package, then I suppose the version changes are fine and the project is OK to delete altogether.

If it were so published, then I'd prefer requiring only 15.something versions of MSBuild, for Visual Studio 2017 compatibility as per Choose the MSBuild API version to reference
.

As I understand it, the main difficulties with running complex code in an MSBuild task are assembly version conflicts and NuGet packaging. And the benefits are support for more complex output parameters (multiple properties, and items with metadata, for subsequent processing in MSBuild) and better integration with the MSBuild logging system (verbosity settings and HelpLink, which MSBuild itself cannot parse from stdout/stderr streams).

There seems to be some progress towards making custom MSBuild tasks easier to create and package: dotnet/msbuild#10733.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Roger. I will leave the MSBuild project here for now and lets deal with what version of nuget package we will need once we succesfully revive it.
m
Just to share my memory so at least 2 of us might remeber in the future :) The original issue I faced years ago when I tried to make this a published nupkg was that MSBild tasks have trouble loading external dll dependencies properly. In our case, it was our own Avro main dll. All compiles and builds fine of course, until you need to load the Task nuget package. Lets hope they need some improvement. I remember there was some discussion back then to create a "fat" msbuild task package which contained the avro dlls, however the final decision was to leave it for now.

<MicrosoftCodeAnalysisVersion>4.11.0</MicrosoftCodeAnalysisVersion>
<MicrosoftCodeAnalysisCSharpVersion>4.11.0</MicrosoftCodeAnalysisCSharpVersion>
<MicrosoftNETTestSdkVersion>17.8.0</MicrosoftNETTestSdkVersion>
<NUnitVersion>3.14.0</NUnitVersion>
<NUnitConsoleRunnerVersion>3.16.3</NUnitConsoleRunnerVersion>
<NUnit3TestAdapterVersion>4.5.0</NUnit3TestAdapterVersion>
<StyleCopAnalyzersVersion>1.1.118</StyleCopAnalyzersVersion>
</PropertyGroup>
</Project>
3 changes: 2 additions & 1 deletion share/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,8 @@ RUN cd /opt ; \
wget https://dot.net/v1/dotnet-install.sh ; \
bash ./dotnet-install.sh --channel "6.0" --install-dir "/opt/dotnet" ; \
bash ./dotnet-install.sh --channel "7.0" --install-dir "/opt/dotnet" ; \
bash ./dotnet-install.sh --channel "8.0" --install-dir "/opt/dotnet" ;
bash ./dotnet-install.sh --channel "8.0" --install-dir "/opt/dotnet" ; \
bash ./dotnet-install.sh --channel "9.0" --install-dir "/opt/dotnet" ;

ENV PATH $PATH:/opt/dotnet

Expand Down
Loading