Module @finos/legend-cube - v0.0.6

@finos/legend-cube

The host-free core of Legend Cube, a canvas-based visual query builder (not Legend DataCube, which is @finos/legend-data-cube). It holds the domain model: precise types and values, schemas, the query graph with its schema inference and validation, the transforms, filters, the Cube IR and its emitter, and the saved-spec codec.

The UI, the stores and the Legend engine adapter live in @finos/legend-cube-builder.

Status: work in progress, version 0.0.x. It has relational tables, data product access points and ingest data sets (both beta), Join, Sort, Group (with its aggregations), Pivot (a column's values into columns, through the engine's pivot(), refused on the databases that can't run it), Filter, Restrict, Rename, Distinct, Drop, Limit, Slice, Concat, Difference, Partition (window functions), Extend (computed columns, typed by the engine) and Filter by Expression (a Pure condition, checked by the engine). The API is not stable before 1.0: src/index.ts re-exports whole modules.

  • The query graph is the only source of truth. A CubeDocument holds the context (the model and the runtime), an immutable Query of QueryNodes and Connections, and presentation settings: the graph shown or not, column widths and number formats, and the grid's drill-down. Every edit makes a new Query.
  • Drill-down: checkDrilldown checks a saved drill-down against a run's columns, and deriveDrilldownLevel derives each level's query from the run's, as Cube's own nodes (a Filter of the expanded keys, a Group, a Sort), so every database workaround applies.
  • Inference: buildSchemasAndValidity gives each node its output schema and its errors.
  • Emitting: QueryEmitter turns a valid query into the Cube IR, a host-free Pure AST (printIR prints it for debugging). A Sort's order is written where it is used: before a Limit, Drop or Slice, and before the run's row limit. The builder's v1/ adapter writes the IR as the engine's protocol JSON. The engine's Pure text is never parsed back.
  • Saving: encodeCubeSpec and decodeCubeSpec (or serializeCubeSpec and parseCubeSpec, for text) write and read the saved spec.
  • Node types: createNodeRegistry() lists them, in palette order. The palette, the context menu and the codec all read it.
Folder Holds
types/ CubeType, the registry of precise primitive types, type families, compatibility
values/ literal values, read and checked per type
schema/ Schema and SchemaColumn, schema diffs
graph/ QueryNode and its base classes, Connection, Query, CubeDocument
inference/ schema inference, which marks incomplete and upstream-invalid nodes; validation helpers; the rows' order through the graph (RowOrder.ts)
nodes/ the node registry; sources/ (relational table, data product access point, ingest data set, and the one-kind rule), transforms/ (Join, Sort, Group and its aggregations, Pivot, Filter, Restrict, Rename, Distinct, Drop, Limit, Slice, Concat, Difference, Partition and its window functions, Extend and its typing, Filter by Expression and its check, the row settings they share, the Join and Concat autofixes), Unknown
filter/ the filter tree, operators by type family, helpers that build filters
messages/ the validation messages, including ERR_INCOMPLETE and ERR_SCHEMAS for incomplete and upstream-invalid nodes
ir/ the Cube IR, an emitter per node type (emitters/), the query emitter that isolates windows with lets, the debug printer
drilldown/ the grid's drill-down: its levels and group-row aggregations, checked against a run, and each level's derived query
presentation/ number column formats, by column or for every number column
spec/ the saved-spec codec, a codec per node type (codecs/), migrations
utils/ assertions and JSON helpers

A cube is saved as JSON, format version 1 (CURRENT_FORMAT_VERSION): formatVersion, name, context (the model, kept as given, and the runtime), query (selected and nodes) and meta. Each node has kind, id, its inputs (left out for a source, which has none) and its own fields.

  • Literal values are typed, with numbers written as strings, so no digit is lost; a node's settings, such as a Limit's size, are JSON numbers. A resolved source keeps a snapshot of its schema; an unresolved one is saved without it, and typed again when the cube is imported.
  • Keys this version doesn't know are kept and written back, and a node of an unknown kind is kept as an Unknown node.
  • A spec is at most 1 MiB (MAX_SPEC_BYTES). One saved by a newer version is read as read-only. There are no migrations yet (CUBE_SPEC_MIGRATIONS), since version 1 is the first.
  • The format is a draft until the Cube store exists: it may still change, with migrations.

This package runs anywhere JavaScript runs: no browser, no Node and no Legend host code. In code outside tests:

  • Only relative imports. The package has no runtime dependencies.
  • Only ECMAScript globals. window, document, fetch, console, setTimeout, structuredClone, URL, TextEncoder and the like fail the build.

Four guards enforce it:

  1. package.json declares no dependencies, and only tooling dev dependencies.
  2. The legendCubeHostFree block of the root ESLint config forbids other imports and flags the common browser globals while you type.
  3. tsconfig.build.json compiles with no ambient types ("types": []).
  4. src/__tests__/LegendCubeHostFree.test.ts checks the dependencies and the imports, and compiles the package against the ECMAScript library alone.

Tests, test utilities and mocks (__tests__, __test-utils__ and __mocks__ folders) are exempt: they may use Node and Jest.

Types come only from resolveCubeType, PrimitiveType.get and new EnumType. That keeps a planned change internal: Cube's types are to move onto legend-graph's metamodel, and the core may then depend on legend-graph's metamodel, but never on its V1_* protocol classes.

  • yarn workspace @finos/legend-cube test runs the tests (test:watch to watch). They run under Node and need no engine.
  • yarn workspace @finos/legend-cube dev, or yarn dev:ts at the root, rebuilds on change.

Enumerations

AggregationFunction
AggregationSetting
ComparisonClass
DifferenceSuffix
EmitRole
FilterOperator
PRIMITIVE_TYPE_PATH
TypeFamily
WindowRankFunction
WindowRowFunction

Classes

BinaryNode
ColumnComparisonFilter
CompositeFilter
Concat
Connection
CubeDocument
CubeLambdaReadError
CubeSpecDecodeError
CubeType
DataProductAccessPointSource
Difference
Distinct
Drop
EnumType
Explode
ExpressionFilter
Extend
Filter
FilterRule
FunctionSource
Group
IngestDatasetSource
Join
Limit
NodeRegistry
NotFilter
OpaqueType
Partition
Pivot
PrimitiveType
Query
QueryEmitter
QueryNode
RelationalTableSource
Rename
Restrict
Schema
ServiceSource
Slice
Sort
SourceNode
UnaryNode
UnknownNode
UnreadableContent
UnsupportedFilter

Interfaces

CheckedDrilldown
ColumnAggregation
ColumnDirection
ColumnFormat
ColumnHint
ColumnWidth
ConcatRestrictFix
CubeContext
CubeDialectWorkarounds
CubeExpressionCodec
CubeLambdaParts
CubeSourceStep
CubeSpecMigration
DataProductAccessPointCoordinates
DifferenceSettings
DrilldownAggregation
EmitContext
ExtendColumn
FunctionCoordinates
FunctionParameter
IngestDatasetCoordinates
IntegerRange
InvalidValue
JoinDuplicateFix
JoinSettings
ModelContext
NodeDefinition
NodeSpecCodec
OrderKey
Origin
PivotAggregation
Presentation
ReadEntries
RelationalTableCoordinates
RelationTypeColumn
RenameMapping
ResolvableSource
SchemaColumnChange
SchemaDiff
ServiceCoordinates
SnapshotColumnRest
SortOrderLoss
SortOrderRemoval

Type Aliases

AggregationUse
DrilldownCell
EnteredValue
ExpressionCheck
ExtendTyping
FilterRuleKind
FilterValue
FilterValueItem
FilterValueShape
IR
IRWithOrigin
JsonValue
LiteralValue
QueryRule
RelationExpr
RowOrder
SourceResolution
ValueProblem
WindowFunction

Variables

AGGREGATION_SETTING_DEFAULTS
BLANK_PLACEHOLDER
CONCAT_CODEC
CONCAT_CONVERTED_COLUMN_BASE
CONCAT_DEFINITION
CONCAT_PORT_LABELS
CUBE_DIALECT_WORKAROUNDS
CUBE_SPEC_MIGRATIONS
CURRENT_FORMAT_VERSION
DATA_PRODUCT_ACCESS_POINT_SOURCE_CODEC
DATA_PRODUCT_ACCESS_POINT_SOURCE_DEFINITION
DEFAULT_PIVOT_AGGREGATION
DIFFERENCE_CODEC
DIFFERENCE_DEFINITION
DISTINCT_CODEC
DISTINCT_PAD_COLUMN
DROP_CODEC
EMPTY_OPERATORS
ERR_CHECKING
ERR_INCOMPLETE
ERR_OTHER
ERR_SCHEMAS
ERR_TYPING
EXPLODE_CODEC
EXPLODE_DEFINITION
EXPRESSION_FILTER_CODEC
EXPRESSION_FILTER_DEFINITION
EXTEND_CODEC
EXTEND_DEFINITION
FILTER_CODEC
FILTER_OPERATOR_DESCRIPTIONS
FILTER_OPERATORS
FUNCTION_SOURCE_CODEC
FUNCTION_SOURCE_DEFINITION
GROUP_CODEC
GROUP_DEFINITION
INGEST_DATASET_SOURCE_CODEC
INGEST_DATASET_SOURCE_DEFINITION
JOIN_CODEC
JOIN_KINDS
JOIN_TYPES
LEFT_KEY_SUFFIX
LIMIT_CODEC
MAX_COLUMN_NAME_LENGTH
MAX_SPEC_BYTES
MESSAGE_AGGREGATION_OUTPUT_NAME_INVALID
MESSAGE_CONCAT_COLUMN_ORDER
MESSAGE_CONDITION_NOT_A_LAMBDA
MESSAGE_DATA_PRODUCT_AFTER_TABLE
MESSAGE_FILTER_UNSUPPORTED
MESSAGE_SERVICE_NOT_READABLE
MESSAGE_TABLE_AFTER_DATA_PRODUCT
NEGATIVE_OPERATORS
NON_LITERAL_PATTERN_DATABASE_TYPE
PARTITION_CODEC
PARTITION_DEFINITION
PATTERN_OPERATORS
PIVOT_AGGREGATION_NAME
PIVOT_CODEC
PIVOT_DEFINITION
PIVOT_MAX_VALUES
PIVOT_UNSUPPORTED_DATABASE_TYPES
RELATIONAL_TABLE_SOURCE_CODEC
RENAME_CODEC
RESTRICT_CODEC
ROW_NUMBER_COLUMN
SAVED_TYPE_KEYS
SERVICE_SOURCE_CODEC
SERVICE_SOURCE_DEFINITION
SET_OPERATORS
SLICE_CODEC
SORT_CODEC
SORT_DIRECTION_DESCRIPTIONS
SORT_DIRECTION_LABELS
VARIANT_DATABASE_TYPES
WINDOW_RANK_FUNCTIONS
WINDOW_ROW_FUNCTIONS

Functions

addDrilldownLevel
aggregationColSpec
areCompatibleTypes
assertRowSetting
buildDrilldownKeyRule
buildJoinSchemaColumns
buildQuickFilterRule
buildSchemasAndValidity
canFixJoinDuplicates
canRenameConcatInput
canRestrictConcatInput
changeFilterColumn
changeFilterOperator
checkDrilldown
checkValue
colSpec
columnAccess
computeRowOrders
copyJson
copyQueryNode
createNodeRegistry
decodeCubeSpec
decodeFilterRule
decodeSavedType
dependsOnDatabaseType
deriveDrilldownLevel
describeRowSetting
diffSchemas
emitAggregation
emitConcat
emitDataProductAccessPointSource
emitDifference
emitDistinct
emitDrop
emitExplode
emitExpressionFilter
emitExtend
emitFilter
emitFunctionSource
emitGroup
emitIngestDatasetSource
emitJoin
emitJoinRelation
emitLimit
emitPartition
emitPivot
emitRelationalTableSource
emitRename
emitRestrict
emitRowNumberRange
emitRowOrder
emitServiceSource
emitSlice
emitSort
emitSortedInput
emitSortKeys
encodeColumnAggregations
encodeCubeSpec
encodeFilterRule
encodeSavedType
encodeSchemaSnapshot
encodeSortKeys
ensureSchemas
findCubeLambdaExtends
findCubeLambdaSources
findLikeFunctionWithBackslash
findLikeFunctionWithNonLiteralPattern
findLikeFunctionWithWildcard
findLostSortOrders
findPatternWildcardOperator
findPrimitiveTypeInfo
fixJoinDuplicates
foldColumnName
freezeColumnAggregation
getAggregationAutoName
getAggregationDisplayName
getAggregationResultType
getAggregationSetting
getAvailableAggregations
getAvailableOperators
getCanonicalNumberText
getCollidingJoinColumns
getColumnFormat
getComparisonClass
getConcatConvertedType
getCubeFunctionName
getDatabaseProblemOfType
getDatabaseProblems
getDefaultDrilldownAggregation
getDefaultOperator
getDialectWorkarounds
getDifferenceOutputNames
getDifferenceResultType
getDrilldownAggregations
getDuplicateJoinColumns
getEmitPatternProblem
getEnginePivotColumnName
getExpressionFilterSignature
getExtendSignature
getFunctionParameterValueError
getIntegerRange
getLeastCommonAncestor
getLiteralKinds
getLiteralText
getMergedJoinKeyType
getNegatedOperator
getNonLiteralPatternProblem
getPivotColumnNames
getRelationalDisplayName
getSameNamedJoinKeys
getTemporaryColumnName
getUtf8ByteLength
getVariantDatabaseProblem
givesNonLiteralPattern
givesPatternWildcard
hashText
hasLikeWildcard
hasNoLikeEscape
hasOnlyKeys
hasPatternWildcard
hasUnescapedPatternCharacter
isAggregationFunctionOf
isAggregationNullable
isColumnAggregation
isColumnDirection
isCubeLambdaSource
isDateOrTimestampType
isDrilldownAggregation
isDrilldownLevelType
isEmptyDrilldown
isExactFloatComparison
isFilterRule
isIncompleteError
isJsonObject
isLiteralValue
isOneParameterLambda
isPivotableType
isPositiveWholeNumber
isPrimitiveType
isRowIndex
isSameCubeIR
isSchemasError
isSortableType
isTypingError
isValidColumnName
keepInputOrder
lambdaJson
letBinding
MESSAGE_AGGREGATION_FUNCTION_DISALLOWS_SETTING
MESSAGE_AGGREGATION_FUNCTION_NEEDS_SORT
MESSAGE_AGGREGATION_FUNCTION_SETTING_INVALID
MESSAGE_CONCAT_COLUMN_COUNT
MESSAGE_CONCAT_COLUMN_NAME
MESSAGE_CONCAT_COLUMN_NOT_CONVERTIBLE
MESSAGE_CONCAT_COLUMN_TYPE
MESSAGE_CONDITION_BACKSLASH
MESSAGE_CONDITION_NOT_BOOLEAN
MESSAGE_CONDITION_NOT_CHECKED
MESSAGE_CONDITION_WILDCARD
MESSAGE_DIFFERENCE_COLUMN_IS_JOIN_COLUMN
MESSAGE_DIFFERENCE_OUTPUT_NAME_INVALID
MESSAGE_DIFFERENT_DATABASES
MESSAGE_DRILLDOWN_AGGREGATION_COLUMN_NOT_IN_OUTPUT
MESSAGE_DRILLDOWN_AGGREGATION_INCOMPATIBLE
MESSAGE_DRILLDOWN_AGGREGATION_LISTED_TWICE
MESSAGE_DRILLDOWN_AGGREGATION_UNKNOWN
MESSAGE_DRILLDOWN_COLUMN_LISTED_TWICE
MESSAGE_DRILLDOWN_COLUMN_NOT_GROUPABLE
MESSAGE_DRILLDOWN_COLUMN_NOT_IN_OUTPUT
MESSAGE_EXPLODE_COLUMN_NOT_VARIANT
MESSAGE_EXPRESSION_NOT_A_LAMBDA
MESSAGE_EXPRESSION_NOT_TYPED
MESSAGE_EXPRESSIONS_NOT_TYPED
MESSAGE_FILTER_OPERATOR_UNSUPPORTED
MESSAGE_FILTER_VALUE_BACKSLASH
MESSAGE_FILTER_VALUE_INVALID
MESSAGE_FILTER_VALUE_OUT_OF_RANGE
MESSAGE_FILTER_VALUE_WILDCARD
MESSAGE_GROUP_COLUMN_NOT_GROUPABLE
MESSAGE_PARAMETER_VALUE_INVALID
MESSAGE_PARAMETER_VALUE_OUT_OF_RANGE
MESSAGE_PARTITION_COLUMN_NOT_PARTITIONABLE
MESSAGE_PATTERN_NOT_LITERAL
MESSAGE_PIVOT_AGGREGATION_COLUMN_TAKEN
MESSAGE_PIVOT_AGGREGATION_UNSUPPORTED
MESSAGE_PIVOT_COLUMN_IS_ROW_COLUMN
MESSAGE_PIVOT_COLUMN_NOT_PIVOTABLE
MESSAGE_PIVOT_ROW_COLUMN_NOT_GROUPABLE
MESSAGE_PIVOT_UNSUPPORTED_DATABASE
MESSAGE_PIVOT_VALUE_INVALID
MESSAGE_PIVOT_VALUE_NOT_A_COLUMN_NAME
MESSAGE_PIVOT_VALUES_DIFFER_ONLY_IN_CASE
MESSAGE_PIVOT_VALUES_TOO_MANY
MESSAGE_SORT_COLUMN_NOT_SORTABLE
MESSAGE_SORT_COLUMNS_CUT
MESSAGE_SORT_COLUMNS_DROPPED
MESSAGE_SORT_ORDER_LOST
MESSAGE_SOURCE_KINDS_MIXED
MESSAGE_VARIANT_UNSUPPORTED_DATABASE
migrateCubeSpec
needsDatabaseType
needsWindowSort
negateFilter
normalizeFilter
originOf
parseCubeSpec
parseValue
pathTo
pickUnknownKeys
planConcatRename
planConcatRestrict
planJoinDuplicateFix
printIR
property
qualifyEnumValue
readColumnAggregations
readCubeQuery
readFilterCondition
readItems
readOptionalFiniteNumber
readOptionalString
readSortKeys
readString
relationalSourcesShareDatabase
removeDrilldownLevel
renameConcatInput
rereadFilterValues
rereadQueryFilterValues
resolveCubeType
restrictConcatInput
rowBound
serializeCubeSpec
setColumnFormat
setDrilldownAggregation
sourcesAreOneKind
splitCubeLambda
stableJsonText
takesAggregationColumn
unqualifyEnumValue
unwrapFilter
validate
validateAggregationFunctionAndColumn
validateAllItems
validateColumnAggregation
validateConcatSchemas
validateGroupColumn
validateJoinKeys
validateNoDuplicateColumns
validatePartitionColumn
validateRenameMapping
validateRowIndex
validateSize
validateSortKey