Analysis API¶

Once a business model is constructed in the Data API, requests to the Analysis API may be made to retrieve metrics or other financial modelling output.

Analysis API supports two types of requests:

Request¶

A basic Analysis request begins with constructing a Network Analysis Stream Request.

Alternatively, an Async Network Analysis Request can be constructed for asynchronous processing. An optional result_callback_url can be submitted with Async Network Analysis Request to receive notifications about request completion. The URL should support HTTP PUT method.

Request can be populated with all of the metrics and analysis output needed by providing more than one Network Analysis Request.

Note that trial_range is an optional parameter of Network Analysis Stream Request or Async Network Analysis Request. When trial_range is not provided as part of the request, the full range from the input ledger files is assumed. The explicit or inferred trial_range over which an analysis was performed is returned as part of the Response.

See Aggregate Merging for information on the optional NetworkAnalysisStreamRequest parameter enable_aggregate_merging.

There are three ways to select a structure for analysis.

A reference to a Networks can be provided in the form of a Network Reference.
A Traversal Query may be specified, which is defined by a series of steps that describe which parts of existing financial structures to use in the analysis.
An S3 URI referring to an InlineNetwork (as an alternative to using the Graphene Data API). In this case, the user serializes an Inline Network as a protobuf message saved to S3.

In most cases, when working with a single financial structure, only one Network Analysis Request will be needed, but many separate financial structures can be analyzed by providing many analysis requests all bundled in the same stream request. Each Network Analysis Request may only refer to one Network, and each may only carry a single Context, so even if only one financial structure is to be analyzed, many analysis requests support analyzing the same one in different contexts.

Context¶

The Context for each analysis can affect many aspects of a financial structure, and so the output may be very different if different contexts are used. Normally, a default context will be used which is often just a simple list containing one path element:

["analyzere"]

In cases where some extra convenience is required, the context supports applying sweeping, in-place changes to the entire financial structure without explicit modifications to the underlying financial model. For example, a custom template context could be added to extend the default context so that all ground-up losses are loaded by a factor of 1.2. The request might then include a context that looks like this:

["analyzere", "custom/twenty_percent_load"]

This makes it easy to reuse the same financial structure for multiple analyses and compare the output. All financial structures can have these changes selectively applied at the time of analysis so that no changes using the Data API are required beforehand. The details of how to use this feature are outside the scope of this introductory documentation.

Metrics¶

YLT metrics are retrieved by providing one or more Distribution Metrics Descriptors with each request. Every descriptor will specify the type of aggregation (AEP/OEP) and one or more return periods or exceedance probability thresholds. Returning the full YLT curve is also supported.

For for information, see Metrics.

Co Metrics¶

Co metrics are retrieved by providing one or more Distribution CoMetrics Descriptors with each request. Every descriptor will specify the type of aggregation (AEP/OEP) and one or more return periods.

One or more component targets is required and specified by the descriptor.

The primary target(s) are specified the same way as any other metric or export. Either the nodes in the network that have no outgoing links are used, or else target_nodes can be specified explicitly at the Network Analysis Request level.

For for information, see Co Metrics.

Exports¶

Exports are written to Amazon S3 (for subsequent use) by including one or more Ledger Export Descriptors with an analysis request. All required Ledger columns are assumed exported, and any additional columns specified will also be included.

By default, the path prefix used for exports is exports/ledgers/ and the full, unique path prefix will be returned with the response(s). The path can be overridden by specifying path_override in the LedgerExportDescriptor and this will cause output to be written to a prefix that begins with exports/custom/ledgers/{path_override}/.

By default, each trial is written to a separate parquet row group. It is convenient when a subsequent read of the file only needs to retrieve a single row group to read an entire trial. However, when the trials are very small the overhead of row group metadata can be significant. User can set optional enable_packed_row_groups parameter of the NetworkAnalysisStreamRequest to True to put multiple trials into the same row group. Total number of records in the group cannot exceed 10000 records. If individual trial has more than 10000 records, it will be recorded as a separate row group.

Concurrent requests may not use the same override. If more than one request uses the same override, only one request will succeed. Likewise, more than one node may not be used to write exports to the same custom path.

Note

It is possible to overwrite custom exports in future requests. Take care to ensure that the output is not used in an inconsistent state.

SQL Queries¶

Ledger data can be queried using SQL supported by AWS Athena. The results are returned inline or exported to S3 similar to Ledger Exports. One or more queries can be performed by providing one or more SQL Query Descriptors.

Response¶

After submitting a Network Analysis Stream Request to , a stream of Network Analysis Stream Replies will be returned almost immediately.

The first reply will be either a Stream Header which indicates there will be a stream of replies coming, unless an error is about to be raised. Every subsequent reply can be either a Terminal Analysis Results or a Status. To handle raised errors, refer to Error Handling.

If an Async Network Analysis Request was submitted, an Async Network Analysis Reply will be returned instead, and the analysis results will be written to Amazon S3 asynchronously. The results_uri field of the Async Network Analysis Reply contains the S3 URI the results will be written to. The status field of the Async Network Analysis Reply will indicate if the async request was queued for processing successfully, or if there was an error. Subsequently, request status will be written to Amazon S3. The status path prefix is analysis_results/{global_request_id}/analysis_status.json.

Any errors encountered during asynchronous request processing will be written to Amazon S3. The error details s3 path prefix is analysis_results/{global_request_id}/error_details, the format is google.rpc.status_pb2.Status.

If warnings are encountered during asynchronous request processing, warning details will be written to Amazon S3. The warning details s3 path prefix is analysis_results/{global_request_id}/warning_details, the format is Status. If there are multiple warnings, each of them will be separated by a newline character in the message field of Status. The analysis_status.json for the request will also have with_warnings set to True if the request completed successfully.

If warnings are encountered during synchronous request processing, up to 100 warnings will be sent as replies in the format Status.

If result_callback_url is specified, a Status message serialized to JSON string will be sent to the URL using HTTP PUT method.

Reply Order¶

No order is guaranteed by the API. In each Terminal Analysis Result, the network_index field describes which Network Analysis Request it is associated with, and the terminal_reference field indicates to which Terminal from the network it is associated. At least one result will be returned for each terminal in each network. In cases where a terminal has Subterminals, more results will be returned (one reply per sub-terminal).

Grouping¶

As part of the Aggregation Descriptor of a request, a series of groupings may be specified as groupby_columns. If they are specified, the grouping of the Distribution Metrics Result will indicate which grouped values the metrics for this result represent.

A common usage might be to specify a grouping by a Peril metadata column. If there are three possible values for Peril, (Hurricane, Windstorm, Earthquake), there will be three different Terminal Analysis Result``s, each with a different ``Distribution Metrics Result where the individual grouping fields indicate which Peril value the associated metrics represent.

Multiple levels of groupings are possible, such that if there are two levels of groupings, with 10 and 3 possible values, respectively, 30 different combinations are possible, so up to 30 separate results should be expected.

Terminals¶

Terminals refer to special components of a given financial structure being analyzed. Every financial structure has at least one Terminal. A terminal is a Node that is not linked to any successive nodes in the financial structure. It may have other nodes connected to it, but it is the “final stop” for all records flowing through the structure.

This is important in processing responses, because in every NetworkAnalysisRequest, there could be multiple terminals in the financial structure. If there are multiple terminals, it is necessary to distinguish them, because their metrics, export and SQL query output will be different. The multiple Terminal Analysis Results that are returned in the stream of replies will refer to specific terminals.

Note

Each descriptor provided in the request will generate one or more results.

Since the descriptors at the request level are associated with the specific Network being analyzed, and since each Network may have more than one terminal, the provided descriptors are evaluated once per terminal rather than once for the entire network. This leads to potentially multiple results per descriptor.

Subterminals¶

Sub-terminals are only a concern when Terminals of a Network in an Analysis Request are specified in ways that cause them to be transfored into multiple terminals in the underlying financial model. The details of using subterminals will be omitted for the purposes of this introductory documentation.

When not relying on advanced Network manipulation that generates subterminals, it is safe to assume all Terminal Analysis Results subterminal_index fields are zero and can be ignored.

gPRC response header¶

A GRPC response header "x-graphene-request-id" is returned with server replies. It contains the request id information of the current request. It can be used to uniquely reference a specific request. It should be recorded by client software as it can be useful when resolving any inquiries.

The request id is in the format of <short request id>::<full request id>, where the short request id is the first 8 characters of full request id.

After getting replies from the analysis api, e.g. replies = analysis_stub.RunAnalysis(...) in the code example, the request id can be extracted in the following way:

request_id = dict(replies.initial_metadata()).get("x-graphene-request-id", "UNDEFINED")