Skip to content

Commit

Permalink
Merge pull request #421 from sbesson/ftp_rocky
Browse files Browse the repository at this point in the history
Migration of the IDR FTP service to Rocky Linux 9
  • Loading branch information
sbesson authored May 9, 2024
2 parents 98edf70 + 4d8b90c commit 50983fa
Show file tree
Hide file tree
Showing 7 changed files with 6 additions and 108 deletions.
30 changes: 0 additions & 30 deletions ansible/files/idrftp-aspera-sh.j2

This file was deleted.

3 changes: 0 additions & 3 deletions ansible/idr-02-services.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,6 @@
### Export services
- import_playbook: idr-export.yml

### Transfer services
- import_playbook: idr-transfer.yml

## Search services
- import_playbook: idr-elasticsearch.yml
- import_playbook: idr-searchengine.yml
25 changes: 0 additions & 25 deletions ansible/idr-ftp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,28 +15,3 @@
anonymous_ftp_pasv_max_port: 32222
# anonymous_ftp_image: openmicroscopy/vsftpd-anonymous-upload:0.2.3

tasks:

- name: ftp | create alternate upload group
become: true
group:
gid: "{{ idrftp_alternate_user_id | default(omit) }}"
name: "{{ idrftp_alternate_user_name | default('ftp-upload') }}"
state: present
system: false

- name: ftp | create alternate upload user
become: true
user:
append: false
createhome: true
group: "{{ idrftp_alternate_user_name | default('ftp-upload') }}"
groups: ""
home: "/data/{{ idrftp_alternate_user_name | default('ftp-upload') }}"
name: "{{ idrftp_alternate_user_name | default('ftp-upload') }}"
# Lock account, it will be manually enabled when needed
password: "!"
state: present
uid: "{{ idrftp_alternate_user_id | default(omit) }}"
# Change to always to always lock the account when this playbook is run
update_password: on_create
22 changes: 0 additions & 22 deletions ansible/idr-transfer.yml

This file was deleted.

2 changes: 1 addition & 1 deletion ansible/openstack-create-ftp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@
# Volumes

- role: ome.openstack_volume_storage
openstack_volume_size: "{{ idr_environment_ftp_size | default(15000) }}"
openstack_volume_size: "{{ idr_environment_ftp_size | default(1000) }}"
openstack_volume_vmname: "{{ idr_environment_idr }}-ftp"
openstack_volume_name: "{{ idr_environment_idr }}-ftp-data"
openstack_volume_device: /dev/vdb
2 changes: 1 addition & 1 deletion ansible/requirements.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
version: 1.1.0

- name: ome.anonymous_ftp
version: 0.2.0
version: 0.2.1

- src: ome.basedeps
version: 1.3.1
Expand Down
30 changes: 4 additions & 26 deletions docs/idr-ftp-s3.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,11 @@
# IDR upload server

The IDR upload VM includes FTP and S3 services for handling data submissions.
The IDR upload VM includes a FTP service for handling data submissions especially
in the early stage where sample data requires some testing.


## IDR FTP server
## IDR FTP service

The IDR FTP server runs in Docker, and only allows [passive anonymous write-only uploads](https://github.com/ome/ansible-role-anonymous-ftp/).
The IDR FTP service runs in Docker, and only allows [passive anonymous write-only uploads](https://github.com/ome/ansible-role-anonymous-ftp/).
The server listens on port `21`, with data connections on ports `32022-32222`.
Incoming uploads will appear on the server under `/data/idrftp-incoming/`.


## IDR S3 upload server

The IDR S3 submission server is co-located with the IDR FTP server.
The IDR S3 server is actually a proxy through to the backend S3 filestore provided by EBI, and uses Minio's support for proxying S3 whilst overlaying its own authentication.

Details of how this works are in the [`ome.minio-s3-gateway` Ansible role](https://github.com/ome/ansible-role-minio-s3-gateway).

Users are created and removed using the `/usr/bin/minio-user.sh` script.
For example, to add a new user `user-test`:

sudo minio-user.sh add user-test

The S3 access and secret key will be printed to stdout, and can be passed to the submitter who will be able to write and read to `https://idr-ftp.openmicroscopy.org/idr-upload/user-test/` using an S3 client.

To remove a user run:

sudo minio-user.sh remove user-test

This only deletes the user credentials, it does not delete any data.

Run the script without arguments for full help.

0 comments on commit 50983fa

Please sign in to comment.