.. index:: Loss Set, Financial Structures; Loss Set, Financial Structures; Ledger
.. _loss-set:

Loss Set / Ledger
==================================

A Loss Set is a prerequisite for use of other financial structures. Invoking this template is done to
provide a reference to the input that will be processed in Graphene. Defining multiple Loss Sets for
analysis is supported (and common).

Once a Ledger is defined according to :ref:`ledger-format` or :ref:`advanced-ledger`, the S3 location where it is
stored will be referenced by the Loss Set template so that this data may be included in a larger financial model.

Suppose the following ledger has been created in S3:

.. code-block:: text

    s3://example_bucket/uploads/ledgers/example_ledger_upload/
      |-- ground_up_loss.parquet.1of4
      |-- ground_up_loss.parquet.2of4
      |-- ground_up_loss.parquet.3of4
      |-- ground_up_loss.parquet.4of4

The following template would be used to reference this ledger in a financial model:

.. code-block:: json

    {
        "_schema": "LossSet_1.0",
        "path": "s3://example_bucket/uploads/ledgers/example_ledger_upload/"
    }


Structure
---------

The general structure a Loss Set in Graphene is:

.. code-block:: json

    {
        "_schema": "LossSet_1.0",
        "path": "s3://example_bucket/uploads/ledgers/example_ledger_upload/",
        "occurrence_key_column": "EventId",
        "currency": "USD"
    }

Parameters
----------

The parameters are defined as follows:

+---------------------------+----------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|           Parameter Name  | Required |    Type    |                                                                                                 Description                                                           |
+===========================+==========+============+=======================================================================================================================================================================+
| ``path``                  | Yes      | ``string`` | The unescaped S3 key prefix or full S3 key that represents a complete ledger. This path may be relative to the bucket root or may be absolute.                        |
+---------------------------+----------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``occurrence_key_column`` | No       | ``string`` | Required when distinct events can occur with the same time value.  Refer to :ref:`Occurrence Key <occurrence-key>` for more information                               |
+---------------------------+----------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ``currency``              | No       | ``string`` | The currency in which the input currency values are defined. Defaults to the :ref:`base currency <base-currency>` if not set.                                         |
+---------------------------+----------+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+

.. note:: Avoid S3 keys containing special characters as described in the `S3 User Guide <https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html>`_
    with the exception of delimiting ``/`` characters.

Partitioned ledgers
-------------------

The examples above assume ledgers are partitioned into multiple files, which requires that the common key prefix of
those files is used to define a reference to the ledger. This is often recommended for performance reasons.

However, when it is convenient, a single parquet file may be used to represent a ledger. In this case, the Loss Set
structure would be defined with the full file path as shown:

.. code-block:: json

    {
        "_schema": "LossSet_1.0",
        "path": "s3://example_bucket/uploads/ledgers/misc/a_parquet_file.parquet"
    }

.. _S3_paths:

Relative S3 Paths
-----------------

Both relative and absolute s3 paths are supported for the ``path`` parameter. A relative path is any path that does not
begin with ``s3://``. A relative path will be mapped to an absolute path by concatenating the S3 key provided with
the bucket URL. The ``custom:bucket`` JWT claim defines the bucket name for the authenticated user.  Refer to
:ref:`jwt-claims` for details of this and other claims available to authenticated users.

Example:

.. code-block:: json

    {
        "_schema": "LossSet_1.0",
        "path": "uploads/ledgers/example_ledger_upload/"
    }

Assuming a bucket name of ``example_bucket``, the above template is equivalent to:

.. code-block:: json

    {
        "_schema": "LossSet_1.0",
        "path": "s3://example_bucket/uploads/ledgers/example_ledger_upload/"
    }


.. note:: A path beginning with ``/`` is unlikely to be interpreted as intended since ``/`` is a valid character for
    any S3 key. Graphene interprets a leading ``/`` as part of the key.

    i.e.

    ``/uploads/ledgers/example_ledger_upload/`` refers to ``s3://example_bucket//uploads/ledgers/example_ledger_upload/``
    such that the URL contains two slashes after the bucket name.


Special characters
------------------

Paths referenced by the Loss Set **should** avoid using characters like ``&``, ``+``, ``?``, ``%``, etc.  These characters
normally need to be URL-encoded, although S3 and supporting utilities treat them as-is, without encoding or decoding,
for the purpose of defining an object key.  Refer to the
`S3 object key naming guidelines. <https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html>`_

If special characters must be used as part of s3 object keys and prefixes, for legacy reasons, leave them
unencoded when defining a Loss Set or any entity in Graphene that accepts S3 URLs containing them. This leaves
Graphene and S3 do not distinguish special characters like ``#`` so using the query or fragment for something
like ephemeral metadata is only possible if the full URL is used in both systems.

For example:

+-----------------------------------------------------------------------------------------------+
|      S3 Object                                                                                |
+===============================================================================================+
| ``s3://example_bucket/example_path/special_characters_number_sign_%23?special#123.parquet``   |
+-----------------------------------------------------------------------------------------------+


+-----------------------------------------------------------------------------------------------+
|      Graphene  path                                                                           |
+===============================================================================================+
|  ``s3://example_bucket/example_path/special_characters_number_sign_%23?special#123.parquet``  |
+-----------------------------------------------------------------------------------------------+