perf(federation): Ed25519署名に対応する (#13464)

* 1. ed25519キーペアを発行・Personとして公開鍵を送受信 * validate additionalPublicKeys * getAuthUserFromApIdはmainを選ぶ * ✌️ * fix * signatureAlgorithm * set publicKeyCache lifetime * refresh * httpMessageSignatureAcceptable * ED25519_SIGNED_ALGORITHM * ED25519_PUBLIC_KEY_SIGNATURE_ALGORITHM * remove sign additionalPublicKeys signature requirements * httpMessageSignaturesSupported * httpMessageSignaturesImplementationLevel * httpMessageSignaturesImplementationLevel: '01' * perf(federation): Use hint for getAuthUserFromApId (#13470) * Hint for getAuthUserFromApId * とどのつまりこれでいいのか？ * use @misskey-dev/node-http-message-signatures * fix * signedPost, signedGet * ap-request.tsを復活させる * remove digest prerender * fix test? * fix test * add httpMessageSignaturesImplementationLevel to FederationInstance * ManyToOne * fetchPersonWithRenewal * exactKey * ✌️ * use const * use gen-key-pair fn. from '@misskey-dev/node-http-message-signatures' * update node-http-message-signatures * fix * @misskey-dev/[email protected] * getAuthUserFromApIdでupdatePersonの頻度を増やす * cacheRaw.date * use requiredInputs #13464 (comment) * update @misskey-dev/node-http-message-signatures * clean up * err msg * fix(backend): fetchInstanceMetadataのLockが永遠に解除されない問題を修正 Co-authored-by: まっちゃとーにゅ <[email protected]> * fix httpMessageSignaturesImplementationLevel validation * fix test * fix * comment * comment * improve test * fix * use Promise.all in genRSAAndEd25519KeyPair * refreshAndprepareEd25519KeyPair * refreshAndfindKey * commetn * refactor public keys add * digestプリレンダを復活させる RFC実装時にどうするか考える * fix, async * fix * !== true * use save * Deliver update person when new key generated (not tested) #13464 (comment) * 循環参照で落ちるのを解消？ * fix? * Revert "fix?" This reverts commit 0082f6f. * a * logger * log * change logger * 秘密鍵の変更は、フラグではなく鍵を引き回すようにする * addAllKnowingSharedInboxRecipe * nanka meccha kaeta * delivre * キャッシュ有効チェックはロック取得前に行う * @misskey-dev/[email protected] * PrivateKeyPem * getLocalUserPrivateKey * fix test * if * fix ap-request * update node-http-message-signatures * fix type error * update package * fix type * update package * retry no key * @misskey-dev/[email protected] * fix type error * log keyid * logger * db-resolver * JSON.stringify * HTTP Signatureがなかったり使えなかったりしそうな場合にLD Signatureを活用するように * inbox-delayed use actor if no signature * ユーザーとキーの同一性チェックはhostの一致にする * log signature parse err * save array * とりあえずtryで囲っておく * fetchPersonWithRenewalでエラーが起きたら古いデータを返す * use transactionalEntityManager * fix spdx * @misskey-dev/[email protected] * add comment * fix * publicKeyに配列が入ってもいいようにする #13950 * define additionalPublicKeys * fix * merge fix * refreshAndprepareEd25519KeyPair → refreshAndPrepareEd25519KeyPair * remove gen-key-pair.ts * defaultMaxListeners = 512 * Revert "defaultMaxListeners = 512" This reverts commit f2c412c. * genRSAAndEd25519KeyPairではキーを直列に生成する? * maxConcurrency: 8 * maxConcurrency: 16 * maxConcurrency: 8 * Revert "genRSAAndEd25519KeyPairではキーを直列に生成する?" This reverts commit d0aada5. * maxWorkers: '90%' * Revert "maxWorkers: '90%'" This reverts commit 9e0a93f. * e2e/timelines.tsで個々のテストに対するtimeoutを削除, maxConcurrency: 32 * better error handling of this.userPublickeysRepository.delete * better comment * set result to keypairEntityCache * deliverJobConcurrency: 16, deliverJobPerSec: 1024, inboxJobConcurrency: 4 * inboxJobPerSec: 64 * delete request.headers['host']; * fix * // node-fetch will generate this for us. if we keep 'Host', it won't change with redirects! * move delete host * modify comment * modify comment * fix correct → collect * refreshAndfindKey → refreshAndFindKey * modify comment * modify attachLdSignature * getApId, InboxProcessorService * TODO * [skip ci] add CHANGELOG --------- Co-authored-by: MeiMei <[email protected]> Co-authored-by: まっちゃとーにゅ <[email protected]>
misskey-dev · Jul 17, 2024 · 5f88d56 · 5f88d56
1 parent 3331f39
commit 5f88d56
Show file tree

Hide file tree

Showing 52 changed files with 1,092 additions and 690 deletions.
diff --git a/.config/docker_example.yml b/.config/docker_example.yml
@@ -164,12 +164,12 @@ id: 'aidx'
 #clusterLimit: 1
 
 # Job concurrency per worker
-# deliverJobConcurrency: 128
-# inboxJobConcurrency: 16
+# deliverJobConcurrency: 16
+# inboxJobConcurrency: 4
 
 # Job rate limiter
 # deliverJobPerSec: 128
-# inboxJobPerSec: 32
+# inboxJobPerSec: 64
 
 # Job attempts
 # deliverJobMaxAttempts: 12

diff --git a/.config/example.yml b/.config/example.yml
@@ -230,15 +230,15 @@ id: 'aidx'
 #clusterLimit: 1
 
 # Job concurrency per worker
-#deliverJobConcurrency: 128
-#inboxJobConcurrency: 16
+#deliverJobConcurrency: 16
+#inboxJobConcurrency: 4
 #relationshipJobConcurrency: 16
 # What's relationshipJob?:
 #  Follow, unfollow, block and unblock(ings) while following-imports, etc. or account migrations.
 
 # Job rate limiter
-#deliverJobPerSec: 128
-#inboxJobPerSec: 32
+#deliverJobPerSec: 1024
+#inboxJobPerSec: 64
 #relationshipJobPerSec: 64
 
 # Job attempts

diff --git a/.devcontainer/devcontainer.yml b/.devcontainer/devcontainer.yml
@@ -157,12 +157,12 @@ id: 'aidx'
 #clusterLimit: 1
 
 # Job concurrency per worker
-# deliverJobConcurrency: 128
-# inboxJobConcurrency: 16
+# deliverJobConcurrency: 16
+# inboxJobConcurrency: 4
 
 # Job rate limiter
-# deliverJobPerSec: 128
-# inboxJobPerSec: 32
+# deliverJobPerSec: 1024
+# inboxJobPerSec: 64
 
 # Job attempts
 # deliverJobMaxAttempts: 12

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,8 @@
 - Feat: 通報を受けた際、または解決した際に、予め登録した宛先に通知を飛ばせるように(mail or webhook) #13705
 - Feat: ユーザーのアイコン/バナーの変更可否をロールで設定可能に
   - 変更不可となっていても、設定済みのものを解除してデフォルト画像に戻すことは出来ます
+- Feat: 連合に使うHTTP SignaturesがEd25519鍵に対応するように #13464
+  - Ed25519署名に対応するサーバーが増えると、deliverで要求されるサーバーリソースが削減されます
 - Fix: 配信停止したインスタンス一覧が見れなくなる問題を修正
 - Fix: Dockerコンテナの立ち上げ時に`pnpm`のインストールで固まることがある問題
 - Fix: デフォルトテーマに無効なテーマコードを入力するとUIが使用できなくなる問題を修正

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -185,7 +185,7 @@ TODO
 ## Environment Variable
 
 - `MISSKEY_CONFIG_YML`: Specify the file path of config.yml instead of default.yml (e.g. `2nd.yml`).
-- `MISSKEY_WEBFINGER_USE_HTTP`: If it's set true, WebFinger requests will be http instead of https, useful for testing federation between servers in localhost. NEVER USE IN PRODUCTION.
+- `MISSKEY_USE_HTTP`: If it's set true, federation requests (like nodeinfo and webfinger) will be http instead of https, useful for testing federation between servers in localhost. NEVER USE IN PRODUCTION. (was `MISSKEY_WEBFINGER_USE_HTTP`)
 
 ## Continuous integration
 Misskey uses GitHub Actions for executing automated tests.

diff --git a/chart/files/default.yml b/chart/files/default.yml
@@ -178,12 +178,12 @@ id: "aidx"
 #clusterLimit: 1
 
 # Job concurrency per worker
-# deliverJobConcurrency: 128
-# inboxJobConcurrency: 16
+# deliverJobConcurrency: 16
+# inboxJobConcurrency: 4
 
 # Job rate limiter
-# deliverJobPerSec: 128
-# inboxJobPerSec: 32
+# deliverJobPerSec: 1024
+# inboxJobPerSec: 64
 
 # Job attempts
 # deliverJobMaxAttempts: 12

diff --git a/packages/backend/migration/1708980134301-APMultipleKeys.js b/packages/backend/migration/1708980134301-APMultipleKeys.js
@@ -0,0 +1,39 @@
+/*
+ * SPDX-FileCopyrightText: syuilo and misskey-project
+ * SPDX-License-Identifier: AGPL-3.0-only
+ */
+
+export class APMultipleKeys1708980134301 {
+    name = 'APMultipleKeys1708980134301'
+
+    async up(queryRunner) {
+        await queryRunner.query(`DROP INDEX "public"."IDX_171e64971c780ebd23fae140bb"`);
+        await queryRunner.query(`ALTER TABLE "user_keypair" ADD "ed25519PublicKey" character varying(128)`);
+        await queryRunner.query(`ALTER TABLE "user_keypair" ADD "ed25519PrivateKey" character varying(128)`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" DROP CONSTRAINT "FK_10c146e4b39b443ede016f6736d"`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" DROP CONSTRAINT "PK_10c146e4b39b443ede016f6736d"`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" ADD CONSTRAINT "PK_0db6a5fdb992323449edc8ee421" PRIMARY KEY ("userId", "keyId")`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" DROP CONSTRAINT "PK_0db6a5fdb992323449edc8ee421"`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" ADD CONSTRAINT "PK_171e64971c780ebd23fae140bba" PRIMARY KEY ("keyId")`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" ADD CONSTRAINT "UQ_10c146e4b39b443ede016f6736d" UNIQUE ("userId")`);
+        await queryRunner.query(`CREATE INDEX "IDX_10c146e4b39b443ede016f6736" ON "user_publickey" ("userId") `);
+        await queryRunner.query(`ALTER TABLE "user_publickey" ADD CONSTRAINT "FK_10c146e4b39b443ede016f6736d" FOREIGN KEY ("userId") REFERENCES "user"("id") ON DELETE CASCADE ON UPDATE NO ACTION`);
+    }
+
+    async down(queryRunner) {
+        await queryRunner.query(`ALTER TABLE "user_publickey" DROP CONSTRAINT "FK_10c146e4b39b443ede016f6736d"`);
+        await queryRunner.query(`DROP INDEX "public"."IDX_10c146e4b39b443ede016f6736"`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" DROP CONSTRAINT "UQ_10c146e4b39b443ede016f6736d"`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" DROP CONSTRAINT "PK_171e64971c780ebd23fae140bba"`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" ADD CONSTRAINT "PK_0db6a5fdb992323449edc8ee421" PRIMARY KEY ("userId", "keyId")`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" DROP CONSTRAINT "PK_0db6a5fdb992323449edc8ee421"`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" ADD CONSTRAINT "PK_10c146e4b39b443ede016f6736d" PRIMARY KEY ("userId")`);
+        await queryRunner.query(`ALTER TABLE "user_publickey" ADD CONSTRAINT "FK_10c146e4b39b443ede016f6736d" FOREIGN KEY ("userId") REFERENCES "user"("id") ON DELETE CASCADE ON UPDATE NO ACTION`);
+        await queryRunner.query(`ALTER TABLE "user_profile" ALTER COLUMN "followersVisibility" DROP DEFAULT`);
+        await queryRunner.query(`ALTER TABLE "user_profile" ALTER COLUMN "followersVisibility" TYPE "public"."user_profile_followersVisibility_enum_old" USING "followersVisibility"::"text"::"public"."user_profile_followersVisibility_enum_old"`);
+        await queryRunner.query(`ALTER TABLE "user_profile" ALTER COLUMN "followersVisibility" SET DEFAULT 'public'`);
+        await queryRunner.query(`ALTER TABLE "user_keypair" DROP COLUMN "ed25519PrivateKey"`);
+        await queryRunner.query(`ALTER TABLE "user_keypair" DROP COLUMN "ed25519PublicKey"`);
+        await queryRunner.query(`CREATE UNIQUE INDEX "IDX_171e64971c780ebd23fae140bb" ON "user_publickey" ("keyId") `);
+    }
+}
diff --git a/packages/backend/migration/1709242519122-HttpSignImplLv.js b/packages/backend/migration/1709242519122-HttpSignImplLv.js
@@ -0,0 +1,16 @@
+/*
+ * SPDX-FileCopyrightText: syuilo and misskey-project
+ * SPDX-License-Identifier: AGPL-3.0-only
+ */
+
+export class HttpSignImplLv1709242519122 {
+    name = 'HttpSignImplLv1709242519122'
+
+    async up(queryRunner) {
+        await queryRunner.query(`ALTER TABLE "instance" ADD "httpMessageSignaturesImplementationLevel" character varying(16) NOT NULL DEFAULT '00'`);
+    }
+
+    async down(queryRunner) {
+        await queryRunner.query(`ALTER TABLE "instance" DROP COLUMN "httpMessageSignaturesImplementationLevel"`);
+    }
+}
diff --git a/packages/backend/migration/1709269211718-APMultipleKeysFix1.js b/packages/backend/migration/1709269211718-APMultipleKeysFix1.js
@@ -0,0 +1,16 @@
+/*
+ * SPDX-FileCopyrightText: syuilo and misskey-project
+ * SPDX-License-Identifier: AGPL-3.0-only
+ */
+
+export class APMultipleKeys1709269211718 {
+    name = 'APMultipleKeys1709269211718'
+
+    async up(queryRunner) {
+        await queryRunner.query(`ALTER TABLE "user_publickey" DROP CONSTRAINT "UQ_10c146e4b39b443ede016f6736d"`);
+    }
+
+    async down(queryRunner) {
+			await queryRunner.query(`ALTER TABLE "user_publickey" ADD CONSTRAINT "UQ_10c146e4b39b443ede016f6736d" UNIQUE ("userId")`);
+    }
+}
diff --git a/packages/backend/package.json b/packages/backend/package.json
@@ -79,13 +79,13 @@
 		"@fastify/multipart": "8.3.0",
 		"@fastify/static": "7.0.4",
 		"@fastify/view": "9.1.0",
+		"@misskey-dev/node-http-message-signatures": "0.0.10",
 		"@misskey-dev/sharp-read-bmp": "1.2.0",
 		"@misskey-dev/summaly": "5.1.0",
 		"@napi-rs/canvas": "^0.1.53",
 		"@nestjs/common": "10.3.10",
 		"@nestjs/core": "10.3.10",
 		"@nestjs/testing": "10.3.10",
-		"@peertube/http-signature": "1.7.0",
 		"@sentry/node": "8.13.0",
 		"@sentry/profiling-node": "8.13.0",
 		"@simplewebauthn/server": "10.0.0",

diff --git a/packages/backend/src/@types/http-signature.d.ts b/packages/backend/src/@types/http-signature.d.ts
diff --git a/packages/backend/src/const.ts b/packages/backend/src/const.ts
@@ -9,6 +9,11 @@ export const MAX_NOTE_TEXT_LENGTH = 3000;
 export const USER_ONLINE_THRESHOLD = 1000 * 60 * 10; // 10min
 export const USER_ACTIVE_THRESHOLD = 1000 * 60 * 60 * 24 * 3; // 3days
 
+export const REMOTE_USER_CACHE_TTL = 1000 * 60 * 60 * 3; // 3hours
+export const REMOTE_USER_MOVE_COOLDOWN = 1000 * 60 * 60 * 24 * 14; // 14days
+
+export const REMOTE_SERVER_CACHE_TTL = 1000 * 60 * 60 * 3; // 3hours
+
 //#region hard limits
 // If you change DB_* values, you must also change the DB schema.
 

diff --git a/packages/backend/src/core/AccountUpdateService.ts b/packages/backend/src/core/AccountUpdateService.ts
@@ -3,7 +3,8 @@
  * SPDX-License-Identifier: AGPL-3.0-only
  */
 
-import { Inject, Injectable } from '@nestjs/common';
+import { Inject, Injectable, OnModuleInit } from '@nestjs/common';
+import { ModuleRef } from '@nestjs/core';
 import { DI } from '@/di-symbols.js';
 import type { UsersRepository } from '@/models/_.js';
 import type { MiUser } from '@/models/User.js';
@@ -12,30 +13,44 @@ import { RelayService } from '@/core/RelayService.js';
 import { ApDeliverManagerService } from '@/core/activitypub/ApDeliverManagerService.js';
 import { UserEntityService } from '@/core/entities/UserEntityService.js';
 import { bindThis } from '@/decorators.js';
+import type { PrivateKeyWithPem } from '@misskey-dev/node-http-message-signatures';
 
 @Injectable()
-export class AccountUpdateService {
+export class AccountUpdateService implements OnModuleInit {
+	private apDeliverManagerService: ApDeliverManagerService;
 	constructor(
+		private moduleRef: ModuleRef,
+
 		@Inject(DI.usersRepository)
 		private usersRepository: UsersRepository,
 
 		private userEntityService: UserEntityService,
 		private apRendererService: ApRendererService,
-		private apDeliverManagerService: ApDeliverManagerService,
 		private relayService: RelayService,
 	) {
 	}
 
+	async onModuleInit() {
+		this.apDeliverManagerService = this.moduleRef.get(ApDeliverManagerService.name);
+	}
+
 	@bindThis
-	public async publishToFollowers(userId: MiUser['id']) {
+	/**
+	 * Deliver account update to followers
+	 * @param userId user id
+	 * @param deliverKey optional. Private key to sign the deliver.
+	 */
+	public async publishToFollowers(userId: MiUser['id'], deliverKey?: PrivateKeyWithPem) {
 		const user = await this.usersRepository.findOneBy({ id: userId });
 		if (user == null) throw new Error('user not found');
 
 		// フォロワーがリモートユーザーかつ投稿者がローカルユーザーならUpdateを配信
 		if (this.userEntityService.isLocalUser(user)) {
 			const content = this.apRendererService.addContext(this.apRendererService.renderUpdate(await this.apRendererService.renderPerson(user), user));
-			this.apDeliverManagerService.deliverToFollowers(user, content);
-			this.relayService.deliverToRelays(user, content);
+			await Promise.allSettled([
+				this.apDeliverManagerService.deliverToFollowers(user, content, deliverKey),
+				this.relayService.deliverToRelays(user, content, deliverKey),
+			]);
 		}
 	}
 }
diff --git a/packages/backend/src/core/CreateSystemUserService.ts b/packages/backend/src/core/CreateSystemUserService.ts
@@ -7,7 +7,7 @@ import { randomUUID } from 'node:crypto';
 import { Inject, Injectable } from '@nestjs/common';
 import bcrypt from 'bcryptjs';
 import { IsNull, DataSource } from 'typeorm';
-import { genRsaKeyPair } from '@/misc/gen-key-pair.js';
+import { genRSAAndEd25519KeyPair } from '@/misc/gen-key-pair.js';
 import { MiUser } from '@/models/User.js';
 import { MiUserProfile } from '@/models/UserProfile.js';
 import { IdService } from '@/core/IdService.js';
@@ -38,7 +38,7 @@ export class CreateSystemUserService {
 		// Generate secret
 		const secret = generateNativeUserToken();
 
-		const keyPair = await genRsaKeyPair();
+		const keyPair = await genRSAAndEd25519KeyPair();
 
 		let account!: MiUser;
 
@@ -64,9 +64,8 @@ export class CreateSystemUserService {
 			}).then(x => transactionalEntityManager.findOneByOrFail(MiUser, x.identifiers[0]));
 
 			await transactionalEntityManager.insert(MiUserKeypair, {
-				publicKey: keyPair.publicKey,
-				privateKey: keyPair.privateKey,
 				userId: account.id,
+				...keyPair,
 			});
 
 			await transactionalEntityManager.insert(MiUserProfile, {