Add global config file, implement local and remote registry #817

minhqdao · 2022-12-26T19:23:23Z

tl;dr

Support global config file
Support local registry
Query and parse remote registry
Allow namespace and v (version) specifications for dependencies

Global configuration file

This PR adds a global config file where you can define registry options:

[registry]
path = 'abc' # Location of the local registry (do not use with url or cache_path)
url = 'abc' # Can be used to download packages from a custom remote registry
cache_path = 'abc' # Cache location for downloaded remote packages

The default location of the global config file is ~/.local/share/fpm/config.toml on Unix-based systems and %APPDATA%\local\fpm\config.toml on Windows.

Packages will be queried from the official registry if no config file exists or no [registry] subtable or url were specified.

The default cache_path will be set to ~/.local/share/fpm/dependencies on Unix-based systems and %APPDATA%\local\fpm\dependencies on Windows if not specified.

Both absolute and relative paths can be resolved.

Local Registry

Our goal is to have an official registry that acts as a reliable source for your dependencies. If you somehow prefer to store your dependencies locally, e.g. during development or to have your dependencies bundled with your code, you can define a directory that acts as your local registry in the global config file (#321). fpm will then be able to search the local registry for dependencies listed in the manifest that are not git or direct path dependencies. It can search for the newest version as well as find a specific version.

The local registry needs to be structured in a way that fpm is able to find the correct package. A namespace must be provided and the package must be inside a folder that has the version number as its name. The manifest of an example package called "example" (namespace: "abc", v: "0.1.2") must be located at <path>/abc/example/0.1.2/fpm.toml.

Remote Registry

If url isn't specified in the global config file, dependencies will the drawn from the official registry. In case you decide to use your own custom registry, follow these docs to build your api endpoints. You can query the latest package (according to semantic versioning) or request a specific version. Packages, which are found, will be placed in the cache and are available across the system.

Manifest Changes

namespace and v (version) are added as new keys to manifest dependencies. If you want to use the registry (either local and remote), a namespace must be provided for every package. It will allow using different packages that carry the same name. If you specify v, then fpm will look for that particular version. Without v, you will get the latest version.

Follow-ups:

Support entering of a custom path to the global config file via the CLI.
Support entering of a path or url to the registry via the CLI.
Cache git dependencies as well.
Handle constraints.
Clear registry cache with fpm clean --registry-cache.
Support packages in the cache that have the same version.
Document.
Implement fpm publish.

Relevant issues: #551, #814.

@awvwgk @everythingfunctional @gnikit @milancurcic @perazz @henilp105 @arteevraina

…ix tests by adding returns

everythingfunctional

I think this is probably headed in the right direction. Just had a few comments/suggestions.

test/fpm_test/test_registry.f90

src/fpm_filesystem.F90

src/fpm_registry.f90

minhqdao · 2022-12-28T22:40:16Z

Maybe we can get one or two more opinions on this to get it merged soon. I kind of need it to continue working on the local registry. Also this is just the first step of a bigger feature so things might (and most probably will) change on the way. @fortran-lang/fpm

minhqdao · 2022-12-30T22:13:56Z

Will put this on draft, change it up and keep working on it a bit. 🙃

minhqdao · 2023-03-10T00:29:22Z

Ready for review 😅

minhqdao · 2023-03-30T13:44:07Z

Ready for review @henilp105 @arteevraina @perazz

perazz

Mostly stylistic comments but LGTM, thank you @minhqdao!

app/main.f90

perazz · 2023-03-16T13:12:15Z

src/fpm.f90

Again, stylistic changes

I can't see what you are referring to.

perazz · 2023-03-16T13:17:26Z

src/fpm/downloader.f90

+  type downloader_t
+  contains
+    procedure, nopass :: get_pkg_data, get_file, unpack
+  end type


For style consistency with fpm, i'ts a good idea to wrap all end all commands with their names (end type downloader_t, end subroutine get_pkg_data, etc...)

You are right, of course. But I have to say that I really dislike it because it is nothing but duplicated code. It takes longer to write and refactor. I'd like to see what other reviewers have to say before I change all my end commands. It's not that we've otherwise been very consistent, e.g. in terms of the formatting.

src/fpm/downloader.f90

perazz · 2023-03-16T13:24:21Z

src/fpm/downloader.f90

+    json = ptr
+  end
+
+  subroutine get_file(url, tmp_file, error)


I'm in favor of renaming this to web_get or something that makes it clear we're downloading something from the web, e.g. download_file

If you only look at the procedure name, then yes, you are right. But this is a member procedure, meaning that it will, for example, be used as downloader%get_file(). Therefore I think it will always be clear that this procedure downloads a file from the web.

src/fpm_command_line.f90

perazz · 2023-03-16T13:59:39Z

fpm.toml


 [dependencies.M_CLI2]
 git = "https://github.com/urbanjost/M_CLI2.git"
 rev = "7264878cdb1baff7323cc48596d829ccfe7751b8"

+[dependencies.jonquil]


This project seems less active than JSON-Fortran. Maybe please check with @certik which of the JSON packages from fortran-lang authors is more easily compiled with LFortran

I first tried json-fortran but it's not compatible with fpm because of the many preprocessor commands. It doesn't work with FPM_BOOTSTRAP, for example.

src/fpm_settings.f90

perazz · 2023-03-16T14:36:10Z

src/fpm/dependency.f90

      end if
+    else
+      call get_global_settings(global_settings, error)


Does this read the settings file from disk every time a new dependency is built? If so, this operation could be repeated hundreds of time, I'd rather save it as a global variable to be loaded when fpm is started, or pass it as a dummy argument to this subroutine

You are right and this should be changed.

It's been changed here: 0c59079

arteevraina · 2023-04-06T16:16:31Z

src/fpm_settings.f90

+  private
+  public :: fpm_global_settings, get_global_settings, get_registry_settings, official_registry_base_url
+
+  character(*), parameter :: official_registry_base_url = 'https://fpm-registry.onrender.com'


Can we put this base url out side of the source code in some environment variable and read if from there ?

You can specify a custom registry url in the global config file if that is what you mean

Yes, the base url might not be final here, but it is also nothing that should be set by the user. I'm also a bit puzzled whether we should leave it like that. @awvwgk

arteevraina

Changes look fine to me overall. Except the one in which we hardcode the base url to the registry which is subject to change for sure.

# Conflicts: # src/fpm.f90

minhqdao · 2023-04-07T09:49:20Z

Thanks everyone, I'll merge this. Next PRs will be shorter :)

minhqdao added 13 commits December 23, 2022 14:09

Add registry settings

7d42784

Rename set_default_prefix and move to fpm_filesystem

8787dd5

Use function instead of subroutine

baff313

Extract get_registry into its own fpm_registry module

15de7fe

Add test for no config file

f7cf187

Inject path for reproducibility

8d815c4

Improve test

db27755

Use correct function

1bb98d5

Use concatination

24ccb14

Add test for empty file and fix bugs

44a410e

Unify logic

4684acf

Add error handling if custom path could not get resolved, add docs, f…

3c1aff6

…ix tests by adding returns

Rename get_registry to get_registry_settings

ff7429a

everythingfunctional reviewed Dec 26, 2022

View reviewed changes

test/fpm_test/test_registry.f90 Outdated Show resolved Hide resolved

src/fpm_filesystem.F90 Outdated Show resolved Hide resolved

src/fpm_registry.f90 Outdated Show resolved Hide resolved

minhqdao added 4 commits December 27, 2022 00:19

Remove test and fix docs

7c86169

Use path to the folder

4705bfe

Improve error message bc it does not have to be config.toml there

239f908

Put in share folder

2a6bfe0

everythingfunctional approved these changes Dec 27, 2022

View reviewed changes

minhqdao marked this pull request as draft December 30, 2022 22:14

minhqdao added 8 commits December 31, 2022 00:06

Use fpm_global_settings and path

1f4ea48

Rename files separately for easier reviewing :)

073cb4b

Nit comment

044f7e3

Not use share folder on Windows machines

6295ebe

Eliminate temporary variable

b1142c2

Parse path and url registry settings and add tests

281e6a4

Improve naming

eb180e1

Add is_absolute_path to fpm_filesystem with tests

4a25a12

minhqdao requested review from awvwgk and everythingfunctional and removed request for awvwgk March 10, 2023 00:28

Improve safety

13db2fe

minhqdao marked this pull request as draft March 16, 2023 18:25

minhqdao added 2 commits March 28, 2023 20:35

Resolve conflicts

56b846f

Merge branch 'main' into resolve-config-file

83d9a00

perazz approved these changes Mar 30, 2023

View reviewed changes

minhqdao added 3 commits April 2, 2023 14:29

Merge branch 'main' into resolve-config-file

9af6e0d

Updae jonquil

6c065ce

Merge main

c32575f

minhqdao marked this pull request as ready for review April 2, 2023 10:26

minhqdao added 8 commits April 2, 2023 18:11

Use consistent toml style

225a462

Change registry base url

81ba0bb

Use longer names

e1afc82

Use more informative variable names

acf4139

Remove unused import

cd9e5c6

Improve variable names again

29ea77f

Only read global settings once

0c59079

Remove duplication

ed27f9b

arteevraina reviewed Apr 6, 2023

View reviewed changes

arteevraina approved these changes Apr 6, 2023

View reviewed changes

Merge branch 'main' into resolve-config-file

e1eb03b

# Conflicts: # src/fpm.f90

minhqdao merged commit 4ba0bf9 into fortran-lang:main Apr 7, 2023

minhqdao deleted the resolve-config-file branch April 7, 2023 09:49

minhqdao mentioned this pull request May 6, 2023

Add docs for the global config file, registry, manifest changes and publish fortran-lang/fpm-docs#102

Merged

5 tasks

minhqdao mentioned this pull request Jun 18, 2023

Enter custom path to the config file via command line #945

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add global config file, implement local and remote registry #817

Add global config file, implement local and remote registry #817

minhqdao commented Dec 26, 2022 •

edited

Loading

everythingfunctional left a comment

minhqdao commented Dec 28, 2022

minhqdao commented Dec 30, 2022

minhqdao commented Mar 10, 2023

minhqdao commented Mar 30, 2023

perazz left a comment

perazz Mar 16, 2023

minhqdao Apr 2, 2023

perazz Mar 16, 2023

minhqdao Apr 2, 2023

perazz Mar 16, 2023

minhqdao Apr 2, 2023

perazz Mar 16, 2023 •

edited

Loading

minhqdao Apr 2, 2023 •

edited

Loading

perazz Mar 16, 2023

minhqdao Apr 2, 2023

minhqdao Apr 2, 2023

arteevraina Apr 6, 2023

minhqdao Apr 6, 2023 •

edited

Loading

minhqdao Apr 6, 2023

arteevraina left a comment

minhqdao commented Apr 7, 2023

Add global config file, implement local and remote registry #817

Add global config file, implement local and remote registry #817

Conversation

minhqdao commented Dec 26, 2022 • edited Loading

tl;dr

Global configuration file

Local Registry

Remote Registry

Manifest Changes

everythingfunctional left a comment

Choose a reason for hiding this comment

minhqdao commented Dec 28, 2022

minhqdao commented Dec 30, 2022

minhqdao commented Mar 10, 2023

minhqdao commented Mar 30, 2023

perazz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

perazz Mar 16, 2023 • edited Loading

Choose a reason for hiding this comment

minhqdao Apr 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

minhqdao Apr 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arteevraina left a comment

Choose a reason for hiding this comment

minhqdao commented Apr 7, 2023

minhqdao commented Dec 26, 2022 •

edited

Loading

perazz Mar 16, 2023 •

edited

Loading

minhqdao Apr 2, 2023 •

edited

Loading

minhqdao Apr 6, 2023 •

edited

Loading