Course of Huge Quantities of MELT Knowledge
Cisco Observability Platform s designed to ingest and course of huge quantities of MELT (Metrics, Occasions, Logs and Traces) information. It’s constructed on high of open requirements like OpenTelemetry to make sure interoperability.
What units it aside is its provision of extensions, empowering our companions and prospects to tailor each side of its performance to their distinctive wants. Our focus right this moment is unveiling the intricacies of customizations particularly tailor-made for information processing. It’s anticipated that you’ve an understanding of the platform fundamentals, like Versatile Metadata Mannequin (FMM) and resolution improvement. Let’s dive in!
The info processing pipeline has varied phases that result in information storage. As MELT information strikes by the pipeline, it’s processed, reworked, and enriched, and ultimately lands within the information retailer the place it may be queried with Unified Question Language (UQL):
Every stage marked with a gear icon permits customization of particular logic. Moreover, the platform allows the creation of fully customized post-processing logic when information can now not be altered.
To streamline customization whereas sustaining flexibility, we’re embracing a brand new method: workflows, faucets, and plugins, using the CNCF Serverless Workflow specification with JSONata because the default expression language. Since Serverless Workflows are designed utilizing open requirements, we’re extensively using CloudEvents and OpenAPI specs. By leveraging these open requirements, we guarantee compatibility and ease of improvement.
Knowledge processing phases that permit information mutation are referred to as faucets, and their customizations plugins. Every faucet declares an enter and output JSON schema for its plugins. Plugins are anticipated to supply an output that adheres to the faucet’s output schema. A faucet is chargeable for merging outputs from all its plugins and producing a brand new occasion, which is a modified model of an authentic occasion. Faucets can solely be authored by the platform, whereas plugins might be created by any resolution in addition to common customers of the platform.
Workflows are meant for post-processing and thus can solely subscribe to triggers (see under). Workflow use circumstances vary from easy occasion counting to stylish machine studying mannequin inferences. Anybody can creator workflows.
This abstraction permits builders to purpose by way of a single occasion, with out exposing the complexity of the underlying stream processing, and use acquainted nicely documented requirements, each of which decrease the barrier of entry.
Every information processing stage communicates with different phases through occasions, which permits us to decouple customers and producers and seamlessly rearrange the phases ought to the necessity come up.
Every occasion has an related class, which determines whether or not a selected stage can subscribe to or publish that occasion. There are two public classes for data-related occasions:
- information:statement – a class of occasions with publish-only permissions which might be considered side-effects of processing the unique occasion, for instance, an entity derived from useful resource attributes in OpenTelemetry metric packet. Observations are indicated with upward ‘publish’ arrows within the above diagram. Faucets, workflows and plugins can all produce observations. Observations can solely be subscribed to by particular faucets.
- information:set off – subscribe-only occasions which can be emitted after all of the mutations have accomplished. Triggers are indicated with a lightning ‘set off’ icon within the above diagram. Solely workflows (post-processing logic) can subscribe to triggers, and solely particular faucets can publish them.
There are 5 statement occasion sorts within the platform:
- entity.noticed – FMM entity was found whereas processing some information. It may be a brand new entity or an replace to an current entity. Every replace from the identical supply totally replaces the earlier one.
- affiliation.noticed – FMM affiliation was found whereas processing some information. Relying on the cardinality of the affiliation the replace logic differs
- extension.noticed – FMM extension attributes have been found whereas processing some information. A goal entity should exist already.
- measurement.obtained – a measurement occasion which contributes to a selected FMM metric. These measurements can be aggregated right into a metric in Metric aggregation faucet. Aggregation logic relies on the metric’s content material sort.
- occasion.obtained – raises a brand new FMM occasion. This occasion can even be processed by the Occasion processing faucet, identical to externally ingested occasions.
There are 3 set off occasion sorts within the platform, one for every information sort: metric.enriched, occasion.enriched, hint.encriched. All three occasions are emitted from the ultimate ‘Tag enrichment’ faucet.
Every occasion is registered in a platform’s information retailer, in order that they’re simply discoverable. To listing all obtainable occasions, merely use fsoc to question them, i.e., to get all triggers:
fsoc information get --type=contracts:cloudevent --filter="information.class eq 'information:set off'" --layer-type=TENANT
Be aware that every one occasion sorts are versioned to permit for evolution and are certified with platform resolution identifier for isolation. For instance, a totally certified id of measurement.obtained occasion is platform:measurement.obtained.v1
Let’s illustrate the above ideas with a simple instance. Contemplate a workflow designed to depend well being rule violations for Kubernetes workloads and APM providers. The logic of the workflow might be damaged down into a number of steps:
- Subscribe to the set off occasion
- Validate occasion sort and entity relevance
- Publish a measurement occasion counting violations whereas retaining severity
Improvement Instruments
Builders can make the most of varied instruments to help in workflow improvement, akin to web-based editors or IDEs.
It’s essential to make sure expressions and logic are legitimate by unit exams and validation towards outlined schemas.
To assist in that, you possibly can write unit exams by using acknowledged, see an instance for this workflow.
On-line JSONata editor may also be a useful instrument in writing your expressions.
A weblog on workflow testing is coming quickly!
Step by Step Information
Create the workflow DSL
Present a singular identifier and a reputation on your workflow:
id: violations-counter model: '1.0.0' specVersion: '0.8' identify: Violations Counter
Discover the set off occasion
Let’s question our set off utilizing fsoc:
fsoc information get --type=contracts:cloudevent --object-id=platform:occasion.enriched.v1 --layer-type=TENANT
Output:
sort: occasion.enriched.v1 description: Signifies that an occasion was enriched with topology tags dataschema: contracts:jsonSchema/platform:occasion.v1 class: information:set off extensions: - contracts:cloudeventExtension/platform:entitytypes - contracts:cloudeventExtension/platform:supply
Subscribe to the occasion
To subscribe to this occasion, it is advisable to add an occasion definition and occasion state referencing this definition (observe a nature of the reference to the occasion – it have to be certified with its information sort):
occasions: - identify: EventReceived sort: contracts:cloudevent/platform:occasion.enriched.v1 sort: consumed dataOnly: false supply: platform states: - identify: event-received sort: occasion onEvents: - eventRefs: - EventReceived
Examine the occasion
Because the information in workflows is obtained in JSON format, occasion information is described in JSON schema.
Let’s take a look at the JSON schema of this occasion (referenced in dataschema), so you recognize what to anticipate in our workflow:
fsoc information get --type=contracts:jsonSchema --object-id=platform:occasion.v1 --layer-type=TENANT Outcome: $schema: http://json-schema.org/draft-07/schema# title: Occasion $id: occasion.v1 sort: object required: - entities - sort - timestamp properties: entities: sort: array minItems: 1 gadgets: $ref: '#/definitions/EntityReference' sort: $ref: '#/definitions/TypeReference' timestamp: sort: integer description: The timestamp in milliseconds spanId: sort: string description: Span id traceId: sort: string description: Hint id uncooked: sort: string description: The uncooked physique of the occasion report attributes: $ref: '#/definitions/Attributes' tags: $ref: '#/definitions/Tags' additionalProperties: false definitions: Tags: sort: object propertyNames: minLength: 1 maxLength: 256 additionalProperties: sort: string Attributes: sort: object propertyNames: minLength: 1 maxLength: 256 additionalProperties: sort: - string - quantity - boolean - object - array EntityReference: sort: object required: - id - sort properties: id: sort: string sort: $ref: '#/definitions/TypeReference' additionalProperties: false TypeReference: sort: string description: A completely certified FMM sort reference instance: k8s:pod
It’s simple – a single occasion, with a number of entity references. Since dataOnly=false, the payload of the occasion can be enclosed within the information discipline, and extension attributes can even be obtainable to the workflow.
Since we all know the precise FMM occasion sort we’re involved in, you can even question its definition to grasp the attributes that the workflow can be receiving and their semantics:
fsoc information get --type=fmm:occasion --filter="information.identify eq "healthrule.violation" and information.namespace.identify eq "alerting"" --layer-type=TENANT
Validate occasion relevance
You’ll want to make sure that the occasion you obtain is of the right FMM occasion sort, and that referenced entities are related. To do that, you possibly can write an expression in JSONata after which use it in an motion situation:
capabilities:
- identify: checkType
sort: expression
operation: |-
information.sort="alerting:healthrule.violation" and (
'k8s:deployment' in information.entities.sort or
'k8s:statefulset' in information.entities.sort or
'k8s:daemonset' in information.entities.sort or
'k8s:cronjob' in information.entities.sort or
'k8s:managed_job' in information.entities.sort or
'apm:service' in information.entities.sort
)
states:
- identify: event-received
sort: occasion
onEvents:
- eventRefs:
- EventReceived
actions:
- identify: createMeasurement
situation: ${ fn:checkType }
Create and publish an occasion
Let’s discover the measurement statement occasion that it is advisable to publish:
fsoc information get --type=contracts:cloudevent --object-id=platform:measurement.obtained.v1 --layer-type=TENANT
Output:
sort: measurement.obtained.v1 description: Signifies that measurements have been obtained. Measurements are then aggregated right into a metric. dataschema: contracts:jsonSchema/platform:measurement.v1 class: information:statement extensions: - contracts:cloudeventExtension/platform:supply
Now let’s take a look at the measurement schema so you understand how to supply a measurement occasion:
fsoc information get --type=contracts:jsonSchema --object-id=platform:measurement.v1 --layer-type=TENANT
Output:
$schema: http://json-schema.org/draft-07/schema# title: Measurements for a selected metric $id: measurement.v1 sort: object required: - entity - sort - measurements properties: entity: $ref: '#/definitions/EntityReference' sort: $ref: '#/definitions/TypeReference' attributes: $ref: '#/definitions/Attributes' measurements: sort: array minItems: 1 description: Measurement values with timestamp for use for metric computation gadgets: sort: object required: - timestamp oneOf: - required: - intValue - required: - doubleValue properties: timestamp: sort: integer description: The timestamp in milliseconds intValue: sort: integer description: Lengthy worth for use for metric computation. doubleValue: sort: quantity description: Double Measurement worth for use for metric computation. additionalProperties: false additionalProperties: false definitions: Attributes: sort: object propertyNames: minLength: 1 maxLength: 256 additionalProperties: sort: - string - quantity - boolean EntityReference: sort: object required: - id - sort properties: id: sort: string sort: $ref: '#/definitions/TypeReference' additionalProperties: false TypeReference: sort: string description: A completely certified FMM sort identify instance: k8s:pod
Create a measurement
Let’s create one other expression that takes the enter occasion and generates a measurement as per the above schema, and use it in an motion within the occasion state:
capabilities:
...
- identify: createMeasurement
sort: expression
operation: |-
{
'entity': information.entities[0],
'sort': 'sampleworkflow:healthrule.violation.depend',
'attributes': {
'violation_severity': information.attributes.violation_severity
},
'measurements': [
{
'timestamp': data.timestamp,
'intValue':
$exists(data.attributes.'event_details.condition_details.violation_count')? data.attributes.'event_details.condition_details.violation_count': 1
}
]
}
states:
- identify: event-received
sort: occasion
onEvents:
- eventRefs:
- EventReceived
actions:
- identify: createMeasurement
situation: '${ fn:checkType }'
functionRef: createMeasurement
actionDataFilter:
toStateData: '${ measurement }'
Right here we’re preserving the violation_severity attribute from the unique occasion and associating the measurement with the identical entity.
The state execution will lead to a measurement discipline created by createMeasurement motion, however provided that the occasion was attention-grabbing based mostly on the situation.
Be aware that since we’re utilizing a brand new FMM metric sort – sampleworkflow:healthrule.violation.depend – we have to register it through the extension on the goal entity sorts. See full resolution linked under for particulars.
Publish an occasion
The subsequent step is to test if the measurement was certainly created, and produce an occasion if it was. To try this, we are going to use a swap state:
states:
- identify: event-received
sort: occasion
onEvents:
- eventRefs:
- EventReceived
actions:
- identify: createMeasurement
situation: ${ fn:checkType }
functionRef:
refName: createMeasurement
actionDataFilter:
toStateData: ${ measurement }
transition: check-measurement
- identify: check-measurement
sort: swap
dataConditions:
- situation: ${ measurement != null }
finish:
terminate: true
produceEvents:
- eventRef: CreateMeasurement
information: ${ measurement }
defaultCondition:
finish: true
That’s it! You possibly can package deal your workflow in an answer, push your resolution, subscribe to it, and examine the metrics by navigating to the metric explorer at https://<your tenant>.observe.appdynamics.com/discover/cco/metric-explorer
An instance graph sliced by violation_severity
In conclusion, the extensibility of the Cisco Observability Platform empowers builders to tailor information processing to their particular necessities effectively. Whether or not it’s customizing processing logic or implementing complicated workflows, the platform supplies the required instruments and suppleness.
Able to be taught extra? Go to examples repo to discover additional and begin customizing your information processing workflows right this moment.
Share:
