Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MediaProxyから画像を2048x2048で提供出来るようにする #192

Open
Sayamame-beans opened this issue May 22, 2024 · 2 comments
Open
Labels
✨Feature This adds/improves/enhances a feature niri-la This issue or pull request is related to original features of misskey.niri.la

Comments

@Sayamame-beans
Copy link

概要

素で上限を2560にした結果、VRCImageDownloaderが2048までしか対応していないため、Misskeyにリサイズありで上げた画像が読み込めない問題が起きるようになったようです。

The maximum resolution is 2048 × 2048 pixels. Attempting to download larger images will result in an error.

https://creators.vrchat.com/worlds/udon/image-loading/#before-you-begin

Originally posted by @Sayamame-beans in #182 (comment)

Misskey MediaProxy: https://github.com/misskey-dev/media-proxy/blob/master/SPECIFICATION.md

Misskey内の当該処理部分?

private async proxyHandler(request: FastifyRequest<{ Params: { url: string; }; Querystring: { url?: string; }; }>, reply: FastifyReply) {
const url = 'url' in request.query ? request.query.url : 'https://' + request.params.url;
if (typeof url !== 'string') {
reply.code(400);
return;
}
// アバタークロップなど、どうしてもオリジンである必要がある場合
const mustOrigin = 'origin' in request.query;
if (this.config.externalMediaProxyEnabled && !mustOrigin) {
// 外部のメディアプロキシが有効なら、そちらにリダイレクト
reply.header('Cache-Control', 'public, max-age=259200'); // 3 days
const url = new URL(`${this.config.mediaProxy}/${request.params.url || ''}`);
for (const [key, value] of Object.entries(request.query)) {
url.searchParams.append(key, value);
}
return await reply.redirect(
301,
url.toString(),
);
}
// Create temp file
const file = await this.getStreamAndTypeFromUrl(url);
if (file === '404') {
reply.code(404);
reply.header('Cache-Control', 'max-age=86400');
return reply.sendFile('/dummy.png', assets);
}
if (file === '204') {
reply.code(204);
reply.header('Cache-Control', 'max-age=86400');
return;
}
try {
const isConvertibleImage = isMimeImage(file.mime, 'sharp-convertible-image-with-bmp');
const isAnimationConvertibleImage = isMimeImage(file.mime, 'sharp-animation-convertible-image-with-bmp');
if (
'emoji' in request.query ||
'avatar' in request.query ||
'static' in request.query ||
'preview' in request.query ||
'badge' in request.query
) {
if (!isConvertibleImage) {
// 画像でないなら404でお茶を濁す
throw new StatusError('Unexpected mime', 404);
}
}
let image: IImageStreamable | null = null;
if ('emoji' in request.query || 'avatar' in request.query) {
if (!isAnimationConvertibleImage && !('static' in request.query)) {
image = {
data: fs.createReadStream(file.path),
ext: file.ext,
type: file.mime,
};
} else {
const data = (await sharpBmp(file.path, file.mime, { animated: !('static' in request.query) }))
.resize({
height: 'emoji' in request.query ? 128 : 320,
withoutEnlargement: true,
})
.webp(webpDefault);
image = {
data,
ext: 'webp',
type: 'image/webp',
};
}
} else if ('static' in request.query) {
image = this.imageProcessingService.convertSharpToWebpStream(await sharpBmp(file.path, file.mime), 498, 422);
} else if ('preview' in request.query) {
image = this.imageProcessingService.convertSharpToWebpStream(await sharpBmp(file.path, file.mime), 200, 200);
} else if ('badge' in request.query) {
const mask = (await sharpBmp(file.path, file.mime))
.resize(96, 96, {
fit: 'contain',
position: 'centre',
withoutEnlargement: false,
})
.greyscale()
.normalise()
.linear(1.75, -(128 * 1.75) + 128) // 1.75x contrast
.flatten({ background: '#000' })
.toColorspace('b-w');
const stats = await mask.clone().stats();
if (stats.entropy < 0.1) {
// エントロピーがあまりない場合は404にする
throw new StatusError('Skip to provide badge', 404);
}
const data = sharp({
create: { width: 96, height: 96, channels: 4, background: { r: 0, g: 0, b: 0, alpha: 0 } },
})
.pipelineColorspace('b-w')
.boolean(await mask.png().toBuffer(), 'eor');
image = {
data: await data.png().toBuffer(),
ext: 'png',
type: 'image/png',
};
} else if (file.mime === 'image/svg+xml') {
image = this.imageProcessingService.convertToWebpStream(file.path, 2048, 2048);
} else if (!file.mime.startsWith('image/') || !FILE_TYPE_BROWSERSAFE.includes(file.mime)) {
throw new StatusError('Rejected type', 403, 'Rejected type');
}
if (!image) {
if (request.headers.range && file.file && file.file.size > 0) {
const range = request.headers.range as string;
const parts = range.replace(/bytes=/, '').split('-');
const start = parseInt(parts[0], 10);
let end = parts[1] ? parseInt(parts[1], 10) : file.file.size - 1;
if (end > file.file.size) {
end = file.file.size - 1;
}
const chunksize = end - start + 1;
image = {
data: fs.createReadStream(file.path, {
start,
end,
}),
ext: file.ext,
type: file.mime,
};
reply.header('Content-Range', `bytes ${start}-${end}/${file.file.size}`);
reply.header('Accept-Ranges', 'bytes');
reply.header('Content-Length', chunksize);
} else {
image = {
data: fs.createReadStream(file.path),
ext: file.ext,
type: file.mime,
};
}
}
if ('cleanup' in file) {
if ('pipe' in image.data && typeof image.data.pipe === 'function') {
// image.dataがstreamなら、stream終了後にcleanup
image.data.on('end', file.cleanup);
image.data.on('close', file.cleanup);
} else {
// image.dataがstreamでないなら直ちにcleanup
file.cleanup();
}
}
reply.header('Content-Type', image.type);
reply.header('Cache-Control', 'max-age=31536000, immutable');
reply.header('Content-Disposition',
contentDisposition(
'inline',
correctFilename(file.filename, image.ext),
),
);
return image.data;
} catch (e) {
if ('cleanup' in file) file.cleanup();
throw e;
}
}

export function getProxiedImageUrl(imageUrl: string, type?: 'preview' | 'emoji' | 'avatar', mustOrigin = false, noFallback = false): string {
const localProxy = `${url}/proxy`;
if (imageUrl.startsWith(instance.mediaProxy + '/') || imageUrl.startsWith('/proxy/') || imageUrl.startsWith(localProxy + '/')) {
// もう既にproxyっぽそうだったらurlを取り出す
imageUrl = (new URL(imageUrl)).searchParams.get('url') ?? imageUrl;
}
return `${mustOrigin ? localProxy : instance.mediaProxy}/${
type === 'preview' ? 'preview.webp'
: 'image.webp'
}?${query({
url: imageUrl,
...(!noFallback ? { 'fallback': '1' } : {}),
...(type ? { [type]: '1' } : {}),
...(mustOrigin ? { origin: '1' } : {}),
})}`;
}
export function getProxiedImageUrlNullable(imageUrl: string | null | undefined, type?: 'preview'): string | null {
if (imageUrl == null) return null;
return getProxiedImageUrl(imageUrl, type);
}
export function getStaticImageUrl(baseUrl: string): string {
const u = baseUrl.startsWith('http') ? new URL(baseUrl) : new URL(baseUrl, url);
if (u.href.startsWith(`${url}/emoji/`)) {
// もう既にemojiっぽそうだったらsearchParams付けるだけ
u.searchParams.set('static', '1');
return u.href;
}
if (u.href.startsWith(instance.mediaProxy + '/')) {
// もう既にproxyっぽそうだったらsearchParams付けるだけ
u.searchParams.set('static', '1');
return u.href;
}
return `${instance.mediaProxy}/static.webp?${query({
url: u.href,
static: '1',
})}`;
}

詳細

この形に沿うと、恐らく<MisskeyURL>/proxy/?url=画像URL&vrchat=1の形式で最大2048x2048まででリサイズした画像を(一定のキャッシュ期間の上で)生成出来る気がします。
ドライブ画面や、ノート上で画像を右クリックした時などに、このURLを提供する項目を生やすと一応実用は出来るようになると思います。

proxyが普段どう使われてるのか全く認識出来ていないのが不安要素です。

目的

2048x2048を上限にした画像を出力出来るようにして、VRChatへの可搬性を上げる。

@Sayamame-beans Sayamame-beans added ✨Feature This adds/improves/enhances a feature niri-la This issue or pull request is related to original features of misskey.niri.la labels May 22, 2024
@anatawa12
Copy link
Collaborator

やるとしたらmisskey本体とmedia proxyを分離してからにしたい

@Sayamame-beans
Copy link
Author

分離したい了解

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
✨Feature This adds/improves/enhances a feature niri-la This issue or pull request is related to original features of misskey.niri.la
Projects
None yet
Development

No branches or pull requests

2 participants