The possibility of using Pyright to build deducers for Pluto Python application code

[Peefy]

The possibility of using Pyright to build deducers for Python code

Background

Pluto positioning is a tool for static deduce of intent in multilingual application programming, including cloud native, AI applications, etc., without the need for developers to perceive infrastructure such as Docker, Kubernetes, and cloud. Currently, Pluto supports using Typescript to write Pluto applications and deploy them with just one click. We want to extend it to Python, so we need a compiler/analysis/tool/IDE extensions like tsc (The typescript compiler and tools) to help analyze the infra architecture in Python. Pyright is one of the potential candidates.

This article explores the possibility of using Pyright to analyze Pluto Python applications.

Quick Case

import { Router, HttpRequest, HttpResponse } from "@plutolang/pluto";

const router = new Router("router");

router.get("/hello", async (req: HttpRequest): Promise<HttpResponse> => {
  const name = req.query["name"] ?? "Anonym";
  const message = `${name} access at ${Date.now()}`;
  return {
    statusCode: 200,
    body: `Publish a message: ${message}`,
  };
});

This is the simplest example of a pluto typescript application. Let's first write an example of a pluto Python application.

from pluto import Router, HttpRequest, HttpResponse  # Python 3.10+
import datetime

async def hello_handler(request: HttpRequest) -> HttpResponse:
    name = request.query.get("name", "Anonym")
    message = f"{name} access at {datetime.datetime.now()}"
    return HttpResponse(status_code=200, body=f"Publish a message: {message}")

router = Router("router")
router.get("/hello", hello_handler)

The simple code of types Router, HttpRequest of HttpResponse is as follows:

# pyright: strict

from dataclasses import dataclass
from typing import Dict, Coroutine, Any, Callable
from abc import ABC, abstractmethod


class IResource(ABC):
    pass


class IResourceClientApi(ABC):
    pass


class IResourceInfraApi(ABC):
    pass


class IResourceCapturedProps(ABC):
    pass


@dataclass
class HttpRequest:
    query: Dict[str, str]

@dataclass
class HttpResponse:
    status_code: int
    body: str

Handler = Callable[[HttpRequest], Coroutine[Any, Any, HttpResponse]]

class IRouterClientApi(IResourceClientApi):
    @abstractmethod
    def get(self, path: str, req: Handler):
        pass


class IRouterInfraApi(IResourceInfraApi):
    pass


class IRouterCapturedProps(IResourceCapturedProps):
    pass


@dataclass
class Router(IResource, IRouterClientApi, IRouterInfraApi, IRouterCapturedProps):
    name: str

    def __init__(self, name: str):
        self.name = name

    def get(self, path: str, req: Handler):
        # Omit the logic
        pass

Therefore, what we need to do in Python is basically the same as in Typescript, and its basic principle can be referred to here

Why Pyright

• Pyright is written by Typescript and Pluto is written by Typescript.
• Pyright directly relies on typescript and references many of them from the implementation of typescript. The structures of the two projects are also similar
• Both copyright and typescript contain static analysis capabilities, which is consistent with Pluto's positioning.
• The author of Python is currently at Microsoft, and the type systems of Python and TypeScript are borrowing from each other. TSC and Copyright are both maintained by Microsoft, ensuring reliability.
• For a personal reason, KCL's progressive type system to some extent references typescript and python, and there is room for improvement in type derivation and checking.

Why not mypy

See here

Pyright is more suitable for analysis than mypy and implements simple calculation of literal types.

How to use Pyright to analyze Python code

Pyright is a full-featured, standards-based static type checker for Python. It is designed for high performance and can be used with large Python source bases.

The pyright arch is as follows

         languageService
    ｜             |                 |
parser -> binder -> checker
                             |
                    | FileSourceInfo|

• Parer parses code snippets into syntax trees
• Binder analyzes Scope symbol and other information from a syntax tree
• Checker performs type check

Pluto special constructs and methods calling analyze

In pyright, declarations are divided into class and function, and Pluto needs to extend its special class and function declarations based on this, that is, carrying special decorators or implementing a specific interface class or function. This definition is completed in binder, and there is no type information at this time. We need to obtain type information through the analyzer interface function to assist in judgment.

export type Declaration =
    | IntrinsicDeclaration
    | ClassDeclaration
    | SpecialBuiltInClassDeclaration
    | FunctionDeclaration
    | ParameterDeclaration
    | TypeParameterDeclaration
    | TypeAliasDeclaration
    | VariableDeclaration
    | AliasDeclaration;

The workflow is

1. locate all CallNodes in the Pluto Python project using the tree walker function visitCall.
2. Find the declaration leftExpression in the CallNode.
3. If it is a ClassDeclaration. We can get the type information of the ClassDeclaration and judge whether it is a special class
4. Validating the special calling. For example

private _validateIsInstanceCall(node: CallNode) {
    if (
        node.leftExpression.nodeType !== ParseNodeType.Name ||
        (node.leftExpression.value !== 'isinstance' && node.leftExpression.value !== 'issubclass') ||
        node.arguments.length !== 2
    ) {
        return;
    }
    // Omit the other code
}

The core interface is

• TypeEvaluator

export interface TypeEvaluator {
    runWithCancellationToken<T>(token: CancellationToken, callback: () => T): T;

    getType: (node: ExpressionNode) => Type | undefined;
    getTypeResult: (node: ExpressionNode) => TypeResult | undefined;
    getTypeResultForDecorator: (node: DecoratorNode) => TypeResult | undefined;
    getCachedType: (node: ExpressionNode) => Type | undefined;
    getTypeOfExpression: (node: ExpressionNode, flags?: EvaluatorFlags, context?: InferenceContext) => TypeResult;
    getTypeOfAnnotation: (node: ExpressionNode, options?: AnnotationTypeOptions) => Type;
    getTypeOfClass: (node: ClassNode) => ClassTypeResult | undefined;
    getTypeOfFunction: (node: FunctionNode) => FunctionTypeResult | undefined;
    getTypeOfExpressionExpectingType: (node: ExpressionNode, options?: ExpectedTypeOptions) => TypeResult;
    evaluateTypeForSubnode: (subnode: ParseNode, callback: () => void) => TypeResult | undefined;
    evaluateTypesForStatement: (node: ParseNode) => void;
    evaluateTypesForMatchStatement: (node: MatchNode) => void;
    evaluateTypesForCaseStatement: (node: CaseNode) => void;
    evaluateTypeOfParameter: (node: ParameterNode) => void;

    canBeTruthy: (type: Type) => boolean;
    canBeFalsy: (type: Type) => boolean;
    stripLiteralValue: (type: Type) => Type;
    removeTruthinessFromType: (type: Type) => Type;
    removeFalsinessFromType: (type: Type) => Type;

    getExpectedType: (node: ExpressionNode) => ExpectedTypeResult | undefined;
    verifyRaiseExceptionType: (node: RaiseNode) => void;
    verifyDeleteExpression: (node: ExpressionNode) => void;
    validateOverloadedFunctionArguments: (
        errorNode: ExpressionNode,
        argList: FunctionArgument[],
        typeResult: TypeResult<OverloadedFunctionType>,
        typeVarContext: TypeVarContext | undefined,
        skipUnknownArgCheck: boolean,
        inferenceContext: InferenceContext | undefined
    ) => CallResult;
    validateInitSubclassArgs: (node: ClassNode, classType: ClassType) => void;

    isAfterNodeReachable: (node: ParseNode) => boolean;
    isNodeReachable: (node: ParseNode, sourceNode?: ParseNode | undefined) => boolean;
    isAsymmetricAccessorAssignment: (node: ParseNode) => boolean;
    suppressDiagnostics: (node: ParseNode, callback: () => void) => void;
    isSpecialFormClass: (classType: ClassType, flags: AssignTypeFlags) => boolean;

    getDeclarationsForStringNode: (node: StringNode) => Declaration[] | undefined;
    getDeclarationsForNameNode: (node: NameNode, skipUnreachableCode?: boolean) => Declaration[] | undefined;
    getTypeForDeclaration: (declaration: Declaration) => DeclaredSymbolTypeInfo;
    resolveAliasDeclaration: (
        declaration: Declaration,
        resolveLocalNames: boolean,
        options?: ResolveAliasOptions
    ) => Declaration | undefined;
    resolveAliasDeclarationWithInfo: (
        declaration: Declaration,
        resolveLocalNames: boolean,
        options?: ResolveAliasOptions
    ) => DeclarationUtils.ResolvedAliasInfo | undefined;
    getTypeOfIterable: (
        typeResult: TypeResult,
        isAsync: boolean,
        errorNode: ExpressionNode,
        emitNotIterableError?: boolean
    ) => TypeResult | undefined;
    getTypeOfIterator: (
        typeResult: TypeResult,
        isAsync: boolean,
        errorNode: ExpressionNode,
        emitNotIterableError?: boolean
    ) => TypeResult | undefined;
    getGetterTypeFromProperty: (propertyClass: ClassType, inferTypeIfNeeded: boolean) => Type | undefined;
    getTypeOfArgument: (arg: FunctionArgument) => TypeResult;
    markNamesAccessed: (node: ParseNode, names: string[]) => void;
    expandPromotionTypes: (node: ParseNode, type: Type) => Type;
    makeTopLevelTypeVarsConcrete: (type: Type, makeParamSpecsConcrete?: boolean) => Type;
    mapSubtypesExpandTypeVars: (
        type: Type,
        conditionFilter: TypeCondition[] | undefined,
        callback: (expandedSubtype: Type, unexpandedSubtype: Type) => Type | undefined
    ) => Type;
    isTypeSubsumedByOtherType: (type: Type, otherType: Type, allowAnyToSubsume: boolean) => boolean;
    lookUpSymbolRecursive: (node: ParseNode, name: string, honorCodeFlow: boolean) => SymbolWithScope | undefined;
    getDeclaredTypeOfSymbol: (symbol: Symbol) => DeclaredSymbolTypeInfo;
    getEffectiveTypeOfSymbol: (symbol: Symbol) => Type;
    getEffectiveTypeOfSymbolForUsage: (
        symbol: Symbol,
        usageNode?: NameNode,
        useLastDecl?: boolean
    ) => EffectiveTypeResult;
    getInferredTypeOfDeclaration: (symbol: Symbol, decl: Declaration) => Type | undefined;
    getDeclaredTypeForExpression: (expression: ExpressionNode, usage?: EvaluatorUsage) => Type | undefined;
    getFunctionDeclaredReturnType: (node: FunctionNode) => Type | undefined;
    getFunctionInferredReturnType: (type: FunctionType, args?: ValidateArgTypeParams[]) => Type;
    getBestOverloadForArguments: (
        errorNode: ExpressionNode,
        typeResult: TypeResult<OverloadedFunctionType>,
        argList: FunctionArgument[]
    ) => FunctionType | undefined;
    getBuiltInType: (node: ParseNode, name: string) => Type;
    getTypeOfMember: (member: ClassMember) => Type;
    getTypeOfBoundMember(
        errorNode: ExpressionNode,
        objectType: ClassType,
        memberName: string,
        usage?: EvaluatorUsage,
        diag?: DiagnosticAddendum | undefined,
        flags?: MemberAccessFlags,
        selfType?: ClassType | TypeVarType
    ): TypeResult | undefined;
    getBoundMagicMethod: (
        classType: ClassType,
        memberName: string,
        selfType?: ClassType | TypeVarType | undefined,
        diag?: DiagnosticAddendum,
        recursionCount?: number
    ) => FunctionType | OverloadedFunctionType | undefined;
    getTypeOfMagicMethodCall: (
        objType: Type,
        methodName: string,
        argList: TypeResult[],
        errorNode: ExpressionNode,
        inferenceContext: InferenceContext | undefined
    ) => Type | undefined;
    bindFunctionToClassOrObject: (
        baseType: ClassType | undefined,
        memberType: FunctionType | OverloadedFunctionType,
        memberClass?: ClassType,
        treatConstructorAsClassMember?: boolean,
        selfType?: ClassType | TypeVarType,
        diag?: DiagnosticAddendum,
        recursionCount?: number
    ) => FunctionType | OverloadedFunctionType | undefined;
    getCallSignatureInfo: (node: CallNode, activeIndex: number, activeOrFake: boolean) => CallSignatureInfo | undefined;
    getAbstractSymbols: (classType: ClassType) => AbstractSymbol[];
    narrowConstrainedTypeVar: (node: ParseNode, typeVar: TypeVarType) => Type | undefined;

    assignType: (
        destType: Type,
        srcType: Type,
        diag?: DiagnosticAddendum,
        destTypeVarContext?: TypeVarContext,
        srcTypeVarContext?: TypeVarContext,
        flags?: AssignTypeFlags,
        recursionCount?: number
    ) => boolean;
    validateOverrideMethod: (
        baseMethod: Type,
        overrideMethod: FunctionType | OverloadedFunctionType,
        baseClass: ClassType | undefined,
        diag: DiagnosticAddendum,
        enforceParamNames?: boolean
    ) => boolean;
    validateCallArguments: (
        errorNode: ExpressionNode,
        argList: FunctionArgument[],
        callTypeResult: TypeResult,
        typeVarContext?: TypeVarContext,
        skipUnknownArgCheck?: boolean,
        inferenceContext?: InferenceContext
    ) => CallResult;
    validateTypeArg: (argResult: TypeResultWithNode, options?: ValidateTypeArgsOptions) => boolean;
    assignTypeToExpression: (
        target: ExpressionNode,
        type: Type,
        isTypeIncomplete: boolean,
        srcExpr: ExpressionNode
    ) => void;
    assignClassToSelf: (destType: ClassType, srcType: ClassType, assumedVariance: Variance) => boolean;
    getBuiltInObject: (node: ParseNode, name: string, typeArguments?: Type[]) => Type;
    getTypedDictClassType: () => ClassType | undefined;
    getTupleClassType: () => ClassType | undefined;
    getObjectType: () => Type;
    getNoneType: () => Type;
    getUnionClassType(): Type;
    getTypingType: (node: ParseNode, symbolName: string) => Type | undefined;
    inferReturnTypeIfNecessary: (type: Type) => void;
    inferTypeParameterVarianceForClass: (type: ClassType) => void;
    assignTypeArguments: (
        destType: ClassType,
        srcType: ClassType,
        diag: DiagnosticAddendum | undefined,
        destTypeVarContext: TypeVarContext | undefined,
        srcTypeVarContext: TypeVarContext | undefined,
        flags: AssignTypeFlags,
        recursionCount: number
    ) => boolean;
    reportMissingTypeArguments: (node: ExpressionNode, type: Type, flags: EvaluatorFlags) => Type;

    isFinalVariable: (symbol: Symbol) => boolean;
    isFinalVariableDeclaration: (decl: Declaration) => boolean;
    isExplicitTypeAliasDeclaration: (decl: Declaration) => boolean;

    addInformation: (message: string, node: ParseNode, range?: TextRange) => Diagnostic | undefined;
    addUnusedCode: (node: ParseNode, textRange: TextRange) => void;
    addUnreachableCode: (node: ParseNode, textRange: TextRange) => void;
    addDeprecated: (message: string, node: ParseNode) => void;

    addDiagnostic: (
        rule: DiagnosticRule,
        message: string,
        node: ParseNode,
        range?: TextRange
    ) => Diagnostic | undefined;
    addDiagnosticForTextRange: (
        fileInfo: AnalyzerFileInfo,
        diagLevel: DiagnosticLevel,
        rule: DiagnosticRule | '',
        message: string,
        range: TextRange
    ) => Diagnostic | undefined;

    printType: (type: Type, options?: PrintTypeOptions) => string;
    printSrcDestTypes: (srcType: Type, destType: Type) => { sourceType: string; destType: string };
    printFunctionParts: (type: FunctionType, extraFlags?: PrintTypeFlags) => [string[], string];

    getTypeCacheEntryCount: () => number;
    disposeEvaluator: () => void;
    useSpeculativeMode: <T>(speculativeNode: ParseNode | undefined, callback: () => T) => T;
    isSpeculativeModeInUse: (node: ParseNode | undefined) => boolean;
    setTypeResultForNode: (node: ParseNode, typeResult: TypeResult, flags?: EvaluatorFlags) => void;

    checkForCancellation: () => void;
    printControlFlowGraph: (
        flowNode: FlowNode,
        reference: CodeFlowReferenceExpressionNode | undefined,
        callName: string,
        logger: ConsoleInterface
    ) => void;
    printTypeVarContext: (typeVarContext: TypeVarContext) => void;
}

• Class Type Details

interface ClassDetails {
    name: string;
    fullName: string;
    moduleName: string;
    fileUri: Uri;
    flags: ClassTypeFlags;
    typeSourceId: TypeSourceId;
    baseClasses: Type[];
    mro: Type[];
    declaration?: ClassDeclaration | SpecialBuiltInClassDeclaration | undefined;
    declaredMetaclass?: ClassType | UnknownType | undefined;
    effectiveMetaclass?: ClassType | UnknownType | undefined;
    fields: SymbolTable;
    typeParameters: TypeVarType[];
    typeVarScopeId?: TypeVarScopeId | undefined;
    docString?: string | undefined;
    deprecatedMessage?: string | undefined;
    dataClassEntries?: DataClassEntry[] | undefined;
    dataClassBehaviors?: DataClassBehaviors | undefined;
    typedDictEntries?: TypedDictEntries | undefined;
    inheritedSlotsNames?: string[];
    localSlotsNames?: string[];

    // A cache of protocol classes (indexed by the class full name)
    // that have been determined to be compatible or incompatible
    // with this class. We use "object" here to avoid a circular dependency.
    // It's actually a map of ProtocolCompatibility objects.
    protocolCompatibility?: object;

    // Transforms to apply if this class is used as a metaclass
    // or a base class.
    classDataClassTransform?: DataClassBehaviors | undefined;

    // Indicates that one or more type parameters has an
    // autovariance, so variance must be inferred.
    requiresVarianceInference?: boolean;

    // A cached value that indicates whether an instance of this class
    // is hashable (i.e. does not override "__hash__" with None).
    isInstanceHashable?: boolean;
}

A typical analysis function.

    private _validateConditionalIsBool(node: ExpressionNode) {
        const operandType = this._evaluator.getType(node);
        if (!operandType) {
            return;
        }

        let isTypeBool = true;
        const diag = new DiagnosticAddendum();
        this._evaluator.mapSubtypesExpandTypeVars(operandType, /* conditionFilter */ undefined, (expandedSubtype) => {
            if (isAnyOrUnknown(expandedSubtype)) {
                return undefined;
            }

            // If it's a bool (the common case), we're good.
            if (isClassInstance(expandedSubtype) && ClassType.isBuiltIn(expandedSubtype, 'bool')) {
                return undefined;
            }

            // Invoke the __bool__ method on the type.
            const boolReturnType = this._evaluator.getTypeOfMagicMethodCall(
                expandedSubtype,
                '__bool__',
                [],
                node,
                /* inferenceContext */ undefined
            );

            if (!boolReturnType || isAnyOrUnknown(boolReturnType)) {
                return undefined;
            }

            if (isClassInstance(boolReturnType) && ClassType.isBuiltIn(boolReturnType, 'bool')) {
                return undefined;
            }

            // All other types are problematic.
            isTypeBool = false;

            diag.addMessage(
                LocAddendum.conditionalRequiresBool().format({
                    operandType: this._evaluator.printType(expandedSubtype),
                    boolReturnType: this._evaluator.printType(boolReturnType),
                })
            );

            return undefined;
        });

        if (!isTypeBool) {
            this._evaluator.addDiagnostic(
                DiagnosticRule.reportGeneralTypeIssues,
                LocMessage.conditionalOperandInvalid().format({
                    type: this._evaluator.printType(operandType),
                }) + diag.getString(),
                node
            );
        }
    }

Reference

Pyright

• https://github.com/microsoft/pyright
• https://github.com/python/typeshed

Python type related PEPs

• PEP 484 type hints including generics
• PEP 487 simpler customization of class creation
• PEP 526 syntax for variable annotations
• PEP 544 structural subtyping
• PEP 561 distributing and packaging type information
• PEP 563 postponed evaluation of annotations
• PEP 570 position-only parameters
• PEP 585 type hinting generics in standard collections
• PEP 586 literal types
• PEP 589 typed dictionaries
• PEP 591 final qualifier
• PEP 593 flexible variable annotations
• PEP 604 complementary syntax for unions
• PEP 612 parameter specification variables
• PEP 613 explicit type aliases
• PEP 635 structural pattern matching
• PEP 646 variadic generics
• PEP 647 user-defined type guards
• PEP 655 required typed dictionary items
• PEP 673 Self type
• PEP 675 arbitrary literal strings
• PEP 681 dataclass transform
• PEP 692 TypedDict for kwargs typing
• PEP 695 type parameter syntax
• PEP 696 (draft) type defaults for TypeVarLikes
• PEP 698 override decorator for static typing
• PEP 702 marking deprecations
• PEP 705 (draft) TypedDict: read-only items
• PEP 712 (draft) converter parameter on dataclasses.field
• PEP 728 (draft) TypedDict with typed extra items
• PEP 742 (draft) narrowing types with TypeIs