.. index:: ELT Format, Input; ELT Format
.. _elt-format:

ELT Format
==========

An ELT (Event Loss Table) is a pre-simulated set of Event IDs, their
severity distributions, and metadata. Once simulated, the output will be
in :doc:`Ledger format<ledger_format>`.

Graphene supports the `Apache Parquet`_ open standard, but without partitioning,
for ELT input data. You can load and simulate multiple ELTs in a network, but each
ELT path reference must refer to a single parquet file (rather than a set of partitions).

A :doc:`Loss Set ELT </templates/financial_structures/loss_set_elt>` template is required to load ELT data into a network.


Columns
-------
ELT data stored in S3 can be in four different formats, the required fields for each are listed below:

Basic ELT
~~~~~~~~~

+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| Column Name          | Data Type | Required | Nullable? | Description                                            |
+======================+===========+==========+===========+========================================================+
| EventID              | INT64     | Yes      | No        |                                                        |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| Mean                 | DOUBLE    | Yes      | No        |                                                        |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| Rate                 | DOUBLE    | Yes      | No        |                                                        |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| Metadata Field 1...n | | INT32   | No       | Yes       | Additional metadata fields: e.g. Peril, Location, etc. |
|                      | | INT64   |          |           |                                                        |
|                      | | FLOAT32 |          |           |                                                        |
|                      | | STRING  |          |           |                                                        |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+


ELT with Parametric Secondary Uncertainty
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+--------------------------+-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Column Name              | Data Type | Required | Nullable? | Description                                                                                                                                     |
+==========================+===========+==========+===========+=================================================================================================================================================+
| EventID                  | INT64     | Yes      | No        |                                                                                                                                                 |
+--------------------------+-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Mean                     | DOUBLE    | Yes      | No        |                                                                                                                                                 |
+--------------------------+-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Rate                     | DOUBLE    | Yes      | No        |                                                                                                                                                 |
+--------------------------+-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Correlation              | DOUBLE    | Yes      | No        |                                                                                                                                                 |
+--------------------------+-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| DistributionType         | STRING    | Yes      | No        | Supported: ``Beta``                                                                                                                             |
+--------------------------+-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| `DistributionParameters` |           | Yes      | No        | Expecting the specific columns for the specified Distribution Type. For ``Beta`` DistributionType, expecting ``Alpha`` and ``Beta`` columns.    |
+--------------------------+-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| ExposureScalar           | DOUBLE    | No       | Yes       |                                                                                                                                                 |
+--------------------------+-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Metadata Field 1...n     | | INT32   | No       | Yes       | Additional metadata fields: e.g. Peril, Location, etc.                                                                                          |
|                          | | INT64   |          |           |                                                                                                                                                 |
|                          | | FLOAT32 |          |           |                                                                                                                                                 |
|                          | | STRING  |          |           |                                                                                                                                                 |
+--------------------------+-----------+----------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------+


RMS-Style ELT with Parametric Secondary Uncertainty
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| Column Name          | Data Type | Required | Nullable? | Description                                            |
+======================+===========+==========+===========+========================================================+
| EventID              | INT64     | Yes      | No        |                                                        |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| Mean                 | DOUBLE    | Yes      | No        |                                                        |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| Rate                 | DOUBLE    | Yes      | No        |                                                        |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| StandardDeviationI   | DOUBLE    | Yes      | No        | The independent portion of the standard deviation      |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| StandardDeviationC   | DOUBLE    | Yes      | No        | The correlated portion of the standard deviation       |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| ExpValue             | DOUBLE    | Yes      | No        |                                                        |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+
| Metadata Field 1...n | | INT32   | No       | Yes       | Additional metadata fields: e.g. Peril, Location, etc. |
|                      | | INT64   |          |           |                                                        |
|                      | | FLOAT32 |          |           |                                                        |
|                      | | STRING  |          |           |                                                        |
+----------------------+-----------+----------+-----------+--------------------------------------------------------+


This input format will be mapped to the Generic ELT with Parametric Secondary Uncertainty as follows:

+----------------------+-----------------------------------------------------------------------------------------------+
| Generic ELT          | RMS Style ELT                                                                                 |
+======================+===============================================================================================+
| ``EventID``          | ``EventID``                                                                                   |
+----------------------+-----------------------------------------------------------------------------------------------+
| ``Mean``             | ``Mean``                                                                                      |
+----------------------+-----------------------------------------------------------------------------------------------+
| ``Rate``             | ``Rate``                                                                                      |
+----------------------+-----------------------------------------------------------------------------------------------+
| ``Correlation``      | :math:`\frac{StandardDeviationI+StandardDeviationC}{ExpValue}`                                |
+----------------------+-----------------------------------------------------------------------------------------------+
| ``DistributionType`` | Beta                                                                                          |
+----------------------+-----------------------------------------------------------------------------------------------+
| ``Alpha``            | See :doc:`simulation methodology</templates/financial_structures/elt_simulation_methodology>` |
+----------------------+-----------------------------------------------------------------------------------------------+
| ``Beta``             | See :doc:`simulation methodology</templates/financial_structures/elt_simulation_methodology>` |
+----------------------+-----------------------------------------------------------------------------------------------+
| ``ExposureScalar``   | ``ExpValue``                                                                                  |
+----------------------+-----------------------------------------------------------------------------------------------+


ELT with Empirical Secondary Uncertainty
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+----------------------+----------------------------+----------+-----------+-----------------------------------------------------------------------------------------------------------------+
| Column Name          | Data Type                  | Required | Nullable? | Description                                                                                                     |
+======================+============================+==========+===========+=================================================================================================================+
| EventID              | INT64                      | Yes      | No        |                                                                                                                 |
+----------------------+----------------------------+----------+-----------+-----------------------------------------------------------------------------------------------------------------+
| Mean                 | DOUBLE                     | Yes      | No        |                                                                                                                 |
+----------------------+----------------------------+----------+-----------+-----------------------------------------------------------------------------------------------------------------+
| Rate                 | DOUBLE                     | Yes      | No        |                                                                                                                 |
+----------------------+----------------------------+----------+-----------+-----------------------------------------------------------------------------------------------------------------+
| EmpiricalPMF         | List of DOUBLE tuple pairs | Yes      | No        | e.g., ``[[0.1e6,0], [1e6, 0.1], [5e6, 0.2], [10e6, 0.6], [10e6, 0.1]]`` for a 10% chance of a $100 loss exactly,|
|                      |                            |          |           | 20% chance of a loss in a range (100 .. 5M], 70% chance of a loss in a range (5M .. 10M],                       |
|                      |                            |          |           | and 10% chance of a 10M loss exactly.                                                                           |
+----------------------+----------------------------+----------+-----------+-----------------------------------------------------------------------------------------------------------------+
| Metadata Field 1...n | | INT32                    | No       | Yes       | Additional metadata fields: e.g. Peril, Location, etc.                                                          |
|                      | | INT64                    |          |           |                                                                                                                 |
|                      | | FLOAT32                  |          |           |                                                                                                                 |
|                      | | STRING                   |          |           |                                                                                                                 |
+----------------------+----------------------------+----------+-----------+-----------------------------------------------------------------------------------------------------------------+

Example of EmpiricalPMF:

.. code-block:: json

    [
        [0.1e6, 0],
        [0.1e6, 0.1],
        [5e6, 0.2],
        [10e6, 0.6],
        [10e6, 0.1]
    ]

.. plot::

   import matplotlib.pyplot as plt
   plt.plot([0, 0.1e6, 0.1e6, 5e6, 10e6, 10e6],
            [0, 0, 0.1, 0.3, 0.9, 1.0])
   plt.xlabel('Loss')
   plt.ylabel('Cumulative Probability')

Immutability
------------

.. note:: ELT input is treated as immutable but the system does not enforce this!

Once ELT files (in parquet format) are uploaded (currently only AWS S3 is supported), they must
not be changed. The system does not enforce this. In the event that an ELT needs to change, a
new file or set of files needs to be uploaded and referenced with a unique URL.

.. _Apache Parquet: https://parquet.apache.org/