support BigInt

[sachinraja]

Hey, I saw that you recently added support for Date and was wondering if you could do the same for BigInt (a JS primitive)? I am willing to implement it myself if you agree that it should be added.

If it matters, I'm using the TypeCompiler.

[sinclairzx81]

@sachinraja Hi, thanks for the suggestion.

At this stage, There are two main things preventing it's inclusion of Type.BigInt().

The first is that BigInt cannot be expressed in JSON. Currently the Type.Date() and Type.Uint8Array() types have been included only due to msgpack binary serialization support. So would need to confirm msgpack support for BigInt also.

The second is BigInt constraints cannot be expressed in JSON. For example, the minimum, exclusiveMinimum, maximum, exclusiveMaximum and multipleOf constraints may not be expressible in JSON Schema (unless expressing as string values), which has implication to sharing schematics between systems.

I'll leave this issue open for a while and give it a little bit of thought.
Cheers
S

[sachinraja]

Thanks for the quick response! Perhaps there should be a way of defining certain schema types only for use with the type compiler. I'm not sure what the added complexity of that would be though.

[sinclairzx81]

@sachinraja Hi, just had a quick look various support for bigint this morning, and unfortunately, the data type doesn't appear to be supported in msgpack. It also doesn't seem to be configurable Ajv either (even through custom types)

MsgPack

import { encode, decode } from 'npm:@msgpack/msgpack'

encode({ value: 1000n }) // having this work would be the minimum barrier to entry.

Error

error: Uncaught Error: Unrecognized object: [object BigInt]

There does however appear to be an upstream PR in msgpack to support BigInt here msgpack/msgpack-javascript#211, so it would be worth revisiting Type.BigInt() when msgpack decides to support the data type. I do note that the PR dates back to May, so I assume there are other technicalities with other languages supporting this data type (one would assume the msgpack authors are reluctant to provide serialization support for the type if it can't be properly deserialized across all languages, but this is just a guess)

Ajv

I can't seem to configure Ajv for custom types (which is pretty awkward). The following would be the unsafe type for BigInt.

function schemaOf(schemaOf: string, value: unknown, schema: any) {
  // this logic never gets run as ajv seems to ignore the numeric bigint values
  if(schema[Kind] === 'BigInt') {
     return typeof value === 'bigint'
  } else {
    return false
  }
}

const ajv = new Ajv({}).addKeyword({ type: 'integer', keyword: 'typeOf', validate: schemaOf })

const BigInt = () => Type.Unsafe<bigint>({
  [Kind]: 'BigInt',
  type: 'integer',
  typeOf: 'BigInt'
})

const R = ajv.validate(Type.Object({                
  value: BigInt(),
}), {
  value: 1000, // Ajv ignores bigints
})

console.log(R)

There does appear to be a open issue to support BigInt under the integer type, but this appears to be pending implementation ajv-validator/ajv#1116. Last reply on this thread was in June, so not sure what the status is there either.

Metadata Publishing

I think the inability to serialize type information for bigint is probably going to be the main blocker for adding BigInt to TypeBox. Ideally, it should be possible to do the following (where the serialized schema may be published to remote consumers)

const T = Type.Object({
   value: Type.BigInt({ minimum: 0n, maximum: 1000n })
})

const S = JSON.stringify(T) // error: Uncaught TypeError: Do not know how to serialize a BigInt

// const S = {
//   type: 'object',
//   required: ['value'],
//   properties: {
//      value: { type: 'integer', typeOf: 'BigInt', minimum: 0n, maximum: 1000n }
//   }                                                       ^ error      ^ error
// }

The problem here is that the 0n and 1000n wouldn't be serializable in JSON. I think that if TypeBox was to support BigInt, the caveat would be that schemas using it would only be shareable under binary serialization (for example, using msgpack if supported). This might be ok for some application types that default to binary serialization, but a bit too much of a caveat for most applications (where the schema should be human readable)

Summary

I think given some of the technical limitations in ajv, msgpack and the JSON format, there isn't too much TypeBox can do to include the type at this time. At this stage, there doesn't appear to be a specification (or even consistent functionality in the ecosystem) to handle this type (which is really unfortunate). Unlike Type.Date() and Type.Uint8Array() which can at least be handled via binary serialization (i.e. msgpack) and can be documented as in such a way to imply "these types are supported if using binary serialization over the wire with msgpack", without having at least binary serialization as an option, including bigint is a bit too much of a stretch for TypeBox (even tho it could implemented purely within the TypeCompiler fairly easily). Considerations for the type extend do past pure validation, and do encompass things like how to express the type in a standard-ish way, so there's just a few too many unknowns at this point in time.

I might need to defer BigInt for the time being and just keep an eye on upstream functionality being added elsewhere first. Ill track this issue with the label future and pick it up at some point down the line.

[sinclairzx81]

@sachinraja Just as a follow up, I think the fastest path to getting BigInt supported in TypeBox would probably be to provide custom schema support in the compiler (which is still pending). Currently TypeBox provides support for custom string formats, but I think a similar mechanism could be developed for complete custom schemas (which would enable the ability for end users to create BigInt as a custom type)

Linking Reference Issue #244

[sinclairzx81]

@sachinraja Hiya. Have just updated TypeBox to support Custom types. It's now possible to support non JSON Schema types (such as bigint) by registering the types Kind. Custom types work similar to the string Format module, however you will need to specify a [Kind] property on the schema.

Documentation for Custom Types can be found here

BigInt implementation below.

import { Type, Kind } from '@sinclair/typebox'
import { Custom } from '@sinclair/typebox/custom'
import { TypeCompiler } from '@sinclair/typebox/compiler'

// Register Type with the Custom namespace
Custom.Set('BigInt', value => typeof value === 'bigint')

// Create a Type schema using the [Kind] property
const T = Type.Unsafe<bigint>({ [Kind]: 'BigInt' })

// Compile it
const C = TypeCompiler.Compile(T)

// Check it
console.log(C.Check(100n)) // true

As mentioned, I don't think TypeBox can reasonably add bigint as a core type, but this mechanism should provide alternative way to extend the compiler and value infrastructure to enable more exotic values to be validated.

Will close off this issue for now
Cheers
S

[sachinraja]

Thank you so much for the quick turnaround on that @sinclairzx81! This is perfect!

[sinclairzx81]

@sachinraja Hey all good. Just a quick note though. I've actually carried out a subsequent update on custom types to accept a schema instance as the first argument of the custom validator function. This to allow some flexibility to specify user defined constraints on the custom type (this if you're currently making use of custom types as the signature for the validation callback has changed)

Custom.Set<{ min: bigint, max: bigint }>('BigInt', (schema, value) => {
  return typeof value === 'bigint' && value >= schema.min && value <= schema.max)
})

const T = Type.Object({
  x: Type.Unsafe<bigint>({ [Kind]: 'BigInt', min: -100n, max: 100n }),
  y: Type.Unsafe<bigint>({ [Kind]: 'BigInt', min: -100n, max: 100n }),
  z: Type.Unsafe<bigint>({ [Kind]: 'BigInt', min: -100n, max: 100n }),
})

const R = TypeCompiler.Compile(T).Check({ // true
  x: 50n,
  y: 50n,
  z: 50n,
})

Will let this functionality settle for a little while, but I think it should help close the gap for validation requirements that need to work outside the standard JSON schema specification (currently considering using this for mongo ObjectId and other non-serializable types).

Updates are on 0.25.10

Cheers
S

[NickBeukema]

@sinclairzx81 Has the latest update included BigInt, leaving these custom modifications unnecessary? I'm also seeing the Custom module is either moved or removed. Could you update us with your latest suggestion on handling BigInt + Serialization?

[sinclairzx81]

@NickBeukema Hi,

BigInt was recently added on 0.26.0, as there has been a more formal split between Standard and Extended type builders. The BigInt type was added to the Extended type set as there is an expectation that extended types fall outside of the JSON Schema specification and may not be serializable in JSON (or even other formats), but that it's still important to be able represent non-serializable JavaScript primitives types in the library (even if not supported by the JSON Schema specification)

ExtendedTypeBuilder may be renamed to JavaScriptTypeBuilder in later revisions.

Serialization

Previously, the Uint8Array, Date types were included as they were directly encode-able in formats such as msgpack. BigInt was included for similar reasons, specifically for the CBOR https://datatracker.ietf.org/doc/html/rfc8949 encoding format. However since the split between Standard and Extended type sets, TypeBox is a bit more open to non serializable formats (with the general expectation that everything in Standard is fully JSON Schema compliant (and is therefore metadata publishable), and anything in Extended has the potential to be published through alternative encoding schemes (with caveats)

Generally, TypeBox tries to provide type representation (at a minimum) for baseline JavaScript primitives (and still pays mind to whether or not a type can be serialized). The library still has reservations for common collection types like Map and Set, as these are treated as JS standard library types more than core primitives. Exceptions to the rule exist for Symbol which is non-serializable in common formats, but is still baseline in the regard typeof value === 'symbol'.

Custom / Formats

These were renamed to the following in 0.26.0

• Custom (renamed to import { TypeRegistry } from '@sinclair/typebox')
• Format (renamed to import { FormatRegistry } from '@sinclair/typebox')

You should be able to just rename from Custom to TypeRegistry. Information on this change can be found https://github.com/sinclairzx81/typebox/blob/master/changelog/0.26.0.md#Format-Renamed-to-FormatRegistry

Hope this brings some insights into some of the recent changes in the library, happy to discuss more if you have any follow on questions.
Cheers
S