Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(openapi): adding a command to resolve circular and recursive references #1063

Open
wants to merge 58 commits into
base: next
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
59bb8f2
Adding refs to the list of openapi commands
Nov 11, 2024
d82700a
Code to execute the openapi:refs command
Nov 11, 2024
099a6f1
style: apply Prettier formatting
Nov 11, 2024
95208cf
Add 'openapi:refs' command and update related test snapshots
Nov 11, 2024
6249347
docs(openapi) command description
Nov 14, 2024
0a5f141
fix(openapi) use of a premade solution for processing a file
Nov 14, 2024
03c092c
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Nov 14, 2024
3660917
chore: update package-lock.json
Nov 19, 2024
7ca5197
Merge branch 'next' of https://github.com/readmeio/rdme into oleh/ope…
Nov 19, 2024
597ece5
refactor: align code with recent updates
Nov 19, 2024
0b87569
Merge branch 'oleh/openapi-adding-command-to-solve-circularity-and-re…
Nov 19, 2024
3039ea0
chore: minor correction
Nov 19, 2024
dca7b9f
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Nov 19, 2024
349b9f8
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Nov 19, 2024
88bd34e
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Nov 19, 2024
0a3c4e0
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Nov 19, 2024
0c0027e
test: openapi:refs
Nov 19, 2024
18f7b4e
Merge branch 'oleh/openapi-adding-command-to-solve-circularity-and-re…
Nov 19, 2024
8fcc065
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Nov 20, 2024
9fa0ca3
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Nov 20, 2024
a4b1fc0
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Nov 21, 2024
c0aa486
Merge remote-tracking branch 'origin/next' into oleh/openapi-adding-c…
Nov 22, 2024
d938932
docs: updating documentation after merge
Nov 22, 2024
44571ae
chore: small text change
Nov 22, 2024
3f1795b
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Nov 25, 2024
b01b18a
refactor: align code with recent updates
Nov 25, 2024
1d1c7e2
feat: add check and display appropriate message for files that cannot…
Nov 25, 2024
1a25b65
test: unresolvable files
Nov 25, 2024
751c52a
chore: gaps
Nov 29, 2024
19875c1
feat: processing of circularity that cannot be processed
Nov 29, 2024
43f2c82
test: unprocessable files
Nov 29, 2024
37f4d5b
chore: correction
Nov 29, 2024
e8d2b9a
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Dec 2, 2024
b293e72
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
Dec 2, 2024
e2a0948
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Dec 2, 2024
87bdd5e
refactor: requested changes
Dec 2, 2024
91dbb5a
chore: deleting a file
Dec 2, 2024
6de2193
chore: oops
Dec 2, 2024
a55dd14
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
Dec 2, 2024
ce09500
refactor: requested changes
Dec 2, 2024
b224ec6
fix: `openapi:refs` touch ups (#1103)
kanadgupta Dec 5, 2024
9c037c9
test: schemes without properties
Dec 5, 2024
da5b0ad
Merge remote-tracking branch 'origin' into oleh/openapi-adding-comman…
Dec 6, 2024
9249d2a
chore: minor updates
Dec 6, 2024
dd02080
Revert "chore: minor updates"
Dec 6, 2024
4355f9c
chore: minor updates
Dec 6, 2024
a7d8555
chore: lint
Dec 6, 2024
f010a7d
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Dec 6, 2024
f7e52bf
feat: processing schemas with only (beta)
Dec 7, 2024
8bcf655
Merge branch 'oleh/openapi-adding-command-to-solve-circularity-and-re…
Dec 7, 2024
2667d1b
chore: lint
Dec 7, 2024
1d119f2
chore: cleanup
Dec 8, 2024
e790dec
chore: cleanup
Dec 9, 2024
ae39d1e
test: complex case
Dec 9, 2024
6e65d9f
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Dec 9, 2024
91958bd
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Dec 9, 2024
09b8039
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Dec 10, 2024
4011248
Merge branch 'next' into oleh/openapi-adding-command-to-solve-circula…
olehshh Dec 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions __tests__/__snapshots__/index.test.ts.snap
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you delete this file?

Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@ Related commands
$ rdme openapi:inspect Analyze an OpenAPI/Swagger definition for various
OpenAPI and ReadMe feature usage.
$ rdme openapi:reduce Reduce an OpenAPI definition into a smaller subset.
$ rdme openapi:refs The script resolves circular and recursive
references in OpenAPI by replacing them with object
schemas. However, not all circular references can
be resolved. You can run the openapi:inspect
command to identify which references remain
unresolved.
$ rdme openapi:validate Validate your OpenAPI/Swagger definition.
"
`;
Expand Down Expand Up @@ -92,6 +98,12 @@ Related commands
$ rdme openapi:inspect Analyze an OpenAPI/Swagger definition for various
OpenAPI and ReadMe feature usage.
$ rdme openapi:reduce Reduce an OpenAPI definition into a smaller subset.
$ rdme openapi:refs The script resolves circular and recursive
references in OpenAPI by replacing them with object
schemas. However, not all circular references can
be resolved. You can run the openapi:inspect
command to identify which references remain
unresolved.
$ rdme openapi:validate Validate your OpenAPI/Swagger definition.
"
`;
Expand Down Expand Up @@ -140,6 +152,12 @@ Related commands
$ rdme openapi:inspect Analyze an OpenAPI/Swagger definition for various
OpenAPI and ReadMe feature usage.
$ rdme openapi:reduce Reduce an OpenAPI definition into a smaller subset.
$ rdme openapi:refs The script resolves circular and recursive
references in OpenAPI by replacing them with object
schemas. However, not all circular references can
be resolved. You can run the openapi:inspect
command to identify which references remain
unresolved.
$ rdme openapi:validate Validate your OpenAPI/Swagger definition.
"
`;
Expand Down
5 changes: 5 additions & 0 deletions __tests__/lib/__snapshots__/commands.test.ts.snap
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@ exports[`utils > #listByCategory > should list commands by category 1`] = `
"hidden": false,
"name": "openapi:validate",
},
{
"description": "The script resolves circular and recursive references in OpenAPI by replacing them with object schemas. However, not all circular references can be resolved. You can run the openapi:inspect command to identify which references remain unresolved.",
"hidden": false,
"name": "openapi:refs",
},
{
"description": "Alias for \`rdme openapi:validate\` [deprecated].",
"hidden": true,
Expand Down
2 changes: 2 additions & 0 deletions src/cmds/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import OpenAPIConvertCommand from './openapi/convert.js';
import OpenAPICommand from './openapi/index.js';
import OpenAPIInspectCommand from './openapi/inspect.js';
import OpenAPIReduceCommand from './openapi/reduce.js';
import OpenAPISolvingCircularityAndRecursiveness from './openapi/refs.js';
import OpenAPIValidateCommand from './openapi/validate.js';
import ValidateAliasCommand from './validate.js';
import CreateVersionCommand from './versions/create.js';
Expand Down Expand Up @@ -47,6 +48,7 @@ const commands = {
'openapi:inspect': OpenAPIInspectCommand,
'openapi:reduce': OpenAPIReduceCommand,
'openapi:validate': OpenAPIValidateCommand,
'openapi:refs': OpenAPISolvingCircularityAndRecursiveness,

validate: ValidateAliasCommand,
whoami: WhoAmICommand,
Expand Down
235 changes: 235 additions & 0 deletions src/cmds/openapi/refs.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,235 @@
/* eslint-disable no-param-reassign */
import type { ZeroAuthCommandOptions } from '../../lib/baseCommand.js';
import type { OASDocument } from 'oas/types';
import type { IJsonSchema } from 'openapi-types';

import fs from 'node:fs';

// eslint-disable-next-line import/no-extraneous-dependencies
import jsYaml from 'js-yaml';

import analyzeOas from '../../lib/analyzeOas.js';
import Command, { CommandCategories } from '../../lib/baseCommand.js';

interface Options {
spec?: string;
}

type SchemaCollection = Record<string, IJsonSchema>;

class OpenAPISolvingCircularityAndRecursiveness extends Command {
constructor() {
super();
this.command = 'openapi:refs';
this.usage = 'openapi:refs [file]';
this.description =
'The script resolves circular and recursive references in OpenAPI by replacing them with object schemas. However, not all circular references can be resolved. You can run the openapi:inspect command to identify which references remain unresolved.';
this.cmdCategory = CommandCategories.APIS;

this.hiddenArgs = ['spec'];
this.args = [
{
name: 'spec',
type: String,
defaultOption: true,
},
];
}

/**
* Reads and parses an OpenAPI file (JSON or YAML).
* @param {string} filePath - The file path to read.
* @returns {any} The parsed content of the file.
*/
static readOpenApiFile(filePath: string) {
const fileContent = fs.readFileSync(filePath, 'utf8');
return filePath.endsWith('.json') ? JSON.parse(fileContent) : jsYaml.load(fileContent);
}

/**
* Writes OpenAPI data to a file (JSON or YAML).
* @param {string} filePath - The file path to write to.
* @param {OASDocument} data - The data to be written.
*/
static writeOpenApiFile(filePath: string, data: OASDocument) {
const content = filePath.endsWith('.json') ? JSON.stringify(data, null, 2) : jsYaml.dump(data, { noRefs: true }); // Disables YAML anchors
fs.writeFileSync(filePath, content, 'utf8');
}

/**
* Identifies circular references in the OpenAPI document.
* @param {OASDocument} document - The OpenAPI document to analyze.
* @returns {Promise<string[]>} A list of circular reference paths.
*/
static async getCircularRefsFromOas(document: OASDocument): Promise<string[]> {
try {
const analysis = await analyzeOas(document);
const circularRefs = analysis.openapi.circularRefs;
return Array.isArray(circularRefs.locations) ? circularRefs.locations : [];
} catch (error) {
return [`Error analyzing OpenAPI document: ${error}`];
}
}

/**
* Replaces a reference in a schema with an object if it's circular or recursive.
* @param {IJsonSchema} schema - The schema to process.
* @param {string[]} circularRefs - List of circular reference paths.
* @param {string} schemaName - The name of the schema being processed.
* @returns {IJsonSchema} The modified schema or the original.
*/
static replaceRefWithObject(schema: IJsonSchema, circularRefs: string[], schemaName: string): IJsonSchema {
if (schema.$ref) {
const refSchemaName = schema.$ref.split('/').pop() as string;
const isCircular = circularRefs.some(refPath => refPath.includes(refSchemaName));
const isRecursive = schemaName === refSchemaName;

if (schemaName.includes('Ref') && (isCircular || isRecursive)) {
return { type: 'object' } as IJsonSchema;
}
}

return schema;
}

/**
* Recursively replaces references in schemas, transforming circular references to objects.
* @param {IJsonSchema} schema - The schema to process.
* @param {string[]} circularRefs - List of circular reference paths.
* @param {string} schemaName - The name of the schema being processed.
*/
static replaceReferencesInSchema(schema: IJsonSchema, circularRefs: string[], schemaName: string) {
if (schema.type === 'object' && schema.properties) {
for (const prop of Object.keys(schema.properties)) {
let property = JSON.parse(JSON.stringify(schema.properties[prop]));
property = OpenAPISolvingCircularityAndRecursiveness.replaceRefWithObject(property, circularRefs, schemaName);
schema.properties[prop] = property;

// Handle arrays with item references
if (property.type === 'array' && property.items) {
property.items = JSON.parse(JSON.stringify(property.items));
property.items = OpenAPISolvingCircularityAndRecursiveness.replaceRefWithObject(
property.items,
circularRefs,
schemaName,
);
OpenAPISolvingCircularityAndRecursiveness.replaceReferencesInSchema(property.items, circularRefs, schemaName);
}
}
}
}

/**
* Replaces circular references within a collection of schemas.
* @param {SchemaCollection} schemas - Collection of schemas to modify.
* @param {string[]} circularRefs - List of circular reference paths.
*/
static replaceCircularRefs(schemas: SchemaCollection, circularRefs: string[]): void {
const createdRefs = new Set<string>();

function replaceRef(schemaName: string, propertyName: string, refSchemaName: string) {
schemas[schemaName]!.properties![propertyName] = { $ref: `#/components/schemas/${refSchemaName}` } as IJsonSchema;
}

function createRefSchema(originalSchemaName: string, refSchemaName: string) {
if (!createdRefs.has(refSchemaName) && schemas[originalSchemaName]) {
schemas[refSchemaName] = {
type: 'object',
properties: { ...schemas[originalSchemaName].properties },
} as IJsonSchema;
OpenAPISolvingCircularityAndRecursiveness.replaceReferencesInSchema(
schemas[refSchemaName],
circularRefs,
refSchemaName,
);
createdRefs.add(refSchemaName);
}
}

circularRefs.forEach(refPath => {
const refParts = refPath.split('/');
if (refParts.length < 6) {
throw new Error(`Invalid reference path: ${refPath}`);
}

const schemaName = refParts[3];
const propertyName = refParts[5];
const schema = schemas[schemaName];
const property = schema?.properties?.[propertyName];

if (!schema || !property) {
throw new Error(`Schema or property not found for path: ${refPath}`);
}

// Handle references within items in an array
let refSchemaName: string | undefined;
if (
refParts.length > 6 &&
refParts[6] === 'items' &&
property.type === 'array' &&
property.items &&
typeof property.items === 'object'
) {
const itemsRefSchemaName = (property.items as IJsonSchema).$ref?.split('/')[3];
if (itemsRefSchemaName) {
refSchemaName = `${itemsRefSchemaName}Ref`;
property.items = { $ref: `#/components/schemas/${refSchemaName}` } as IJsonSchema;
createRefSchema(itemsRefSchemaName, refSchemaName);
}
} else {
// Handle direct reference
refSchemaName = property.$ref?.split('/')[3];
if (refSchemaName) {
const newRefSchemaName = `${refSchemaName}Ref`;
replaceRef(schemaName, propertyName, newRefSchemaName);
createRefSchema(refSchemaName, newRefSchemaName);
}
}
});
}

/**
* The main execution method for the command.
* @param {ZeroAuthCommandOptions<Options>} opts - Command options.
* @returns {Promise<string>} Result message.
*/
async run(opts: ZeroAuthCommandOptions<Options>): Promise<string> {
await super.run(opts);
const { spec } = opts;
if (!spec) {
return 'File path is required.';
}

const openApiData = OpenAPISolvingCircularityAndRecursiveness.readOpenApiFile(spec);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than using this static method to read the file, can you use our existing prepareOas function? see an example here:

const { preparedSpec, specPath, specType } = await prepareOas(spec, 'openapi:convert', { convertToLatest: true });

const circularRefs = await OpenAPISolvingCircularityAndRecursiveness.getCircularRefsFromOas(openApiData);

if (circularRefs.length === 0) {
return 'The file does not contain circular or recursive references.';
}

if (openApiData.components?.schemas && circularRefs.length > 0) {
OpenAPISolvingCircularityAndRecursiveness.replaceCircularRefs(openApiData.components.schemas, circularRefs);

let remainingCircularRefs = await OpenAPISolvingCircularityAndRecursiveness.getCircularRefsFromOas(openApiData);
let iterationCount = 0;

while (remainingCircularRefs.length > 0 && iterationCount < 5) {
OpenAPISolvingCircularityAndRecursiveness.replaceCircularRefs(
openApiData.components.schemas,
remainingCircularRefs,
);
remainingCircularRefs = remainingCircularRefs.length > 0 ? [] : remainingCircularRefs;
iterationCount += 1;
}

if (iterationCount >= 5) {
return 'Maximum iteration limit reached. Some circular references may remain unresolved.';
}
}

OpenAPISolvingCircularityAndRecursiveness.writeOpenApiFile(spec, openApiData);
Copy link
Member

@kanadgupta kanadgupta Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than overwriting the existing file, can you add a prompt (and a corresponding --out flag) so users can specify what the updated file path should be? see here for an example:

prompts.override({
outputPath: opts.out,
});
const promptResults = await promptTerminal([
{
type: 'text',
name: 'outputPath',
message: 'Enter the path to save your converted/bundled API definition to:',
initial: () => {
const extension = path.extname(specPath);
return `${path.basename(specPath).split(extension)[0]}.openapi${extension}`;
},
validate: value => validateFilePath(value),
},
]);
Command.debug(`saving converted/bundled spec to ${promptResults.outputPath}`);
fs.writeFileSync(promptResults.outputPath, JSON.stringify(parsedPreparedSpec, null, 2));

return `Processed and updated ${spec}`;
}
}

export default OpenAPISolvingCircularityAndRecursiveness;