.. index:: Ledger Format; Simplified, Parquet; Columns, Input; Ledger Format
.. _ledger-format:

Ledger Format
=============

A Ledger is what represents the sequence of financial transactions within a scenario
or a set of scenarios. It differs from a Year Event Loss Table (YELT) in that it is
not limited to record losses, but can capture any form of financial transactions
including premium income, fee payments, taxes, collateral deposit and release, etc.
Also, records in a Ledger are not required to be tied to a specific stochastic
event and in fact event information (event ID, region, peril, model, etc.) are
considered metadata associated with a record in a ledger.

Graphene supports the `Apache Parquet`_ open standard for Ledger input data.

Columns
-------

.. note:: Column names are case-sensitive and required columns must match those specified in the table.

+----------------------+--------------------------+----------+-----------+----------------------------------------------------------------------------------------------------------+
| Column Name          | Data Type                | Required | Nullable? | Description                                                                                              |
+======================+==========================+==========+===========+==========================================================================================================+
| Trial                | INT64                    | Yes      | No        | The index of the independent scenario of the model, otherwise known as "Year".                           |
+----------------------+--------------------------+----------+-----------+----------------------------------------------------------------------------------------------------------+
| Time                 | DOUBLE                   | Yes      | No        | Time value for when this transaction was recorded. See :ref:`timestamps` for additional details.         |
+----------------------+--------------------------+----------+-----------+----------------------------------------------------------------------------------------------------------+
| Type                 | STRING                   | Yes      | No        | The type of record that the monetary value represents (e.g. "Loss" or "Premium")                         |
+----------------------+--------------------------+----------+-----------+----------------------------------------------------------------------------------------------------------+
| Value                | DOUBLE                   | Yes      | No        | The actual monetary value associated with the transaction.  Losses are negative.                         |
+----------------------+--------------------------+----------+-----------+----------------------------------------------------------------------------------------------------------+
| Metadata Field 1...n | | INT32                  | No       | Yes       | Additional metadata fields: e.g. EventId, Peril, Severity, LineOfBusiness, TIV, Location, etc.           |
|                      | | INT64                  |          |           |                                                                                                          |
|                      | | FLOAT32                |          |           |                                                                                                          |
|                      | | STRING                 |          |           |                                                                                                          |
+----------------------+--------------------------+----------+-----------+----------------------------------------------------------------------------------------------------------+


Data types
----------

When multiple ledger inputs are provided, Graphene expects column data types to be defined identically for columns
with the same name.

.. note:: When multiple ledger inputs are provided, Graphene will not attempt to handle a column that has differing
          types among multiple files. Instead, a user-facing error will be returned.


.. _timestamps:

Timestamps
~~~~~~~~~~

Graphene is agnostic to time units and specific calendar systems.
However, you must use a consistent time reference.  We recommend the
`POSIX time format`_ which represents dates and times as seconds since
January 1st, 1970 midnight UTC. However, you are free to choose whatever time
reference or unit you prefer, as long as it meets these requirements:

* it can be represented as a decimal number
* it can be ordered
* it is used consistently across all objects in the system

For example, you may choose to represent time as "days since the beginning
of the year" (any year) without timezones. In this case, midnight of
January 10 of any year would be represented as timestamp value 9, and noon
on January 10 of any year as timestamp value 9.5. However, note that a time
reference like this does not take into account leap years that have 366
days.

.. index:: Input; Occurrence Key

Occurrence key
~~~~~~~~~~~~~~

One of the metadata fields will be designated as an occurrence key in later API requests to Graphene,
which distinguishes records for different events if they happen to have the same Time value.

.. index:: Input; Currency column

Currency column
~~~~~~~~~~~~~~

One of the metadata fields will be designated as a currency columns in later API requests to Graphene,
which allows to specify different currency for each individual record in the input ledger.

.. index:: Input; Parquet Order, Parquet; Order (simplified)

Order
-----

Records need to be ordered by Trial, Time, and then Occurrence Key.


.. index:: Input; Immutability

Immutability
------------

.. note:: Ledger input is treated as immutable but the system does not enforce this!

Once ledger files (in parquet format) are uploaded (currently only AWS S3 is supported), they must
not be changed. The system does not enforce this. In the event that a ledger needs to change, a
new file or set of files needs to be uploaded and referenced with a unique URL.

.. _Apache Parquet: https://parquet.apache.org/
.. _POSIX time format: https://en.wikipedia.org/wiki/Unix_time