Skip to main content

Datalake exports

EVA is an event-driven platform. Everything from transactions and returns to stock movements and mutations is logged and registered as an event in real-time. Using the data from these events you can create your own custom dashboards and reports.

Pushing data to a data lake​

The most flexible and future-proof way of creating dashboard and reports based on your data, is using a data lake. A data lake is a single store of enterprise data including raw copies of source system data. This setup has multiple benefits over a traditional database or data warehouse:

Scalable Unlike a traditional data warehouse, data lakes offer scalability, typically at lower costs.

Versatile A data lake can store both structured and unstructured data from diverse sources. In other words, it can store XML, logs, multimedia, sensor data, chat, social data, binary, and people data.

Schema Flexibility For traditional schema, you need to have your data in a structured format. Traditional data warehouse products are schema based. But for analytics, this could prove to be a glitch as the data needs to be analyzed in its raw form.

Setting up a data lake, including vendor selection and contracting, is a client responsibility. Of course, New Black can assist in this process. We recommend using a Standard general-purpose v2 Azure storage account.

Setting up your data lake​

There are several kinds of data lake integrations possible. These are all described, with their own specific settings, here below. In order to enable the settings, we do need to enable a plugin behind the scenes. Please reach out to your Strategic Lead.

Azure​

To set up your data lake, make sure you have a regular V2 Azure storage account. EVA needs your account's information to export your data. The following settings are absolutely necessary:

SettingDescription
Azure:DataLake:AccountNameAzure Account name
Azure:DataLake:AccountKeyAzure Account Key
Azure:DataLake:FileSystemThe "container" name in the storage account
Azure:DataLake:FolderDesired folder structure, for example: RAW/EVA will create a RAW folder with an EVA subfolder
Azure:DataLake:ServiceUrlYour Azure Service URL

AWS​

The following settings apply if you make use of AWS data lake. These settings make use of the same cron jobs to trigger the exports (see Config below), regardless of the Azure name.

SettingDescription
AWS:DataLake:RegionThe region where the S3 bucket lives.
AWS:DataLake:S3BucketThe name of the S3 bucket to store the files in.
AWS:DataLake:AccessKey-
AWS:DataLake:SecretKey-
AWS:DataLake:Folder(optional) The folder to store the files in the bucket. If not set, we use the root folder.

Google BigQuery​

The following settings apply if you make use of Google BigQuery data lake. These settings make use of the same cron jobs to trigger the exports (see Config below), regardless of the Azure name.

SettingDescription
GCP:DataLake:BucketThe name of the bucket to store the files in.
GCP:DataLake:CredentialThe credential in JSON format.
GCP:DataLake:Folder(optional) The folder to store the files in the bucket. If not set, we use the root folder.

Data Pack​

EVA exports the following data to your data lake:

  1. Orders
  2. Financial periods
  3. Products
  4. Replenishment products
  5. Stock
  6. Stock mutations
  7. Organization units
  8. General ledgers
  9. Prices
  10. Sales invoices
  11. Users
  12. Fulfilment results - to be manually enabled with DataLake:ExportFulfillmentResults

Event ledgers​

Aside from the above list of events, EVA also facilitates the export of event ledgers. Because you might not want your DataLake to be stuffed with event ledgers, you'll need to explicitly enable this by switching DataLake:ExportEventLedger to true.

Event ledger types
- AuthenticationAttempt
- Login
- Logout
- OrderCreated
- OrderLinesPlaced
- OrderAutoCompleted
- ManualReturnOrderCreated
- PaymentCreated
- PaymentFinalized
- CashDrawerOpened
- CashTransaction
- CashDrawerClosed
- CashDrawerClosedTimeout
- CashDrawerClosingDeviation
- CashDrawerOpeningDeviation
- UserTaskStarted
- UserTaskCompleted
- UserDetachedFromUserTask
- UserTaskIgnored
- DiscountUsed
- ManualDiscountGiven
- InvoiceCreated
- InvoicePrinted
- InvoiceDuplicatePrinted
- InvoiceMailed
- InvoiceDownloaded
- InvoiceArchived
- ReceiptPrinted
- ReceiptDuplicatePrinted
- ReceiptPrintFailed
- ReceiptReprinted
- ReceiptReprintFailed
- MessageTemplateUpdated
- SettingUpdated
- EVAInstanceStarted
- FinancialPeriodClosed
- FinancialReportingPeriodClosed
- FinancialPeriodAuditExported
- FiscalPeriodArchived
- CashExpense
- CashDeposit
- CashCorrection
- CashMovement
- UserAttachedToRole
- UserRemovedFromRole
- FunctionalityAddedToUser
- FunctionalityRemovedFromUser
- FunctionalityAddedToRole
- FunctionalityRemovedFromRole
- OrganizationUnitChanged
- OrganizationUnitCreated
- XReport
- ZReport
- FincialPeriodClosingImpedimentIgnored

Config​

Orders, financial periods and sales invoices are exported by default, you can enable the other exports by setting up the following cron tasks:

  • EVA.DataLake.ExportProducts
  • EVA.DataLake.ExportReplenishmentProducts
  • EVA.DataLake.ExportStock
  • EVA.DataLake.ExportStockMutations
  • EVA.DataLake.ExportOrganizationUnits
  • EVA.DataLake.ExportPriceLists
  • EVA.DataLake.ExportUserStoreRelations

Examples​

{
    "ID": "FinancialPeriodID",
    "FinancialPeriodNumber": null,
    "PreviousFinancialPeriodID": "PreviousFinancialPeriodID",
    "OpeningTime": "2021-06-12T07:19:47.963",
    "ClosingTime": null,
    "AutoClosed": false,
    "Status": "Open",
    "OrganizationUnitID": "OUID",
    "User": {
        "ID": "UserID",
        "Nickname": "UserNickname",
        "EmailAddress": "UserMailAddress",
        "FirstName": "UserFirstName",
        "LastName": "UserLastName",
        "Gender": null,
        "LanguageID": "fr",
        "CountryID": "FR",
        "Salutation": null
    },
    "CashDeposits": [],
    "CashJournals": [{
            "FinancialPeriodID": "FinancialPeriodID",
            "PreviousCashJournalID": "PreviousFinancialPeriodID",
            "PaymentTypeID": "PaymentTypeID",
            "PaymentType": {
                "PaymentMethodID": "PaymentMethodID",
                "Name": "Cash",
                "Code": "CASH",
                "LedgerClassID": null,
                "BackendRelationID": null,
                "Category": "Cash",
                "CashJournalMethod": "OpenAndClose"
            },
            "CurrencyID": "EUR",
            "Type": "Default",
            "OpeningTime": "2021-06-12T07:28:07.453",
            "ClosingTime": "2021-06-12T17:29:54.487",
            "OpeningAmount": 491.8800,
            "ClosingAmount": 490.8800,
            "OpeningDeviation": 0.0000,
            "ClosingDeviation": null,
            "StationID": "StationID",
            "DeviceID": "DeviceID",
            "OpeningDetailsObject": {
                "Coins": {
                    "1": 2,
                    "2": 7,
                    "0.01": 3,
                    "0.02": 0,
                    "0.05": 3,
                    "0.1": 7,
                    "0.2": 0,
                    "0.5": 0
                },
                "BankNotes": {
                    "5": 1,
                    "10": 17,
                    "20": 15,
                    "50": 0,
                    "100": 0,
                    "200": 0,
                    "500": 0
                },
                "CashDeposits": {}
            },
            "ClosingDetailsObject": {
                "Coins": {
                    "1": 1,
                    "2": 7,
                    "0.01": 3,
                    "0.02": 0,
                    "0.05": 3,
                    "0.1": 7,
                    "0.2": 0,
                    "0.5": 0
                },
                "BankNotes": {
                    "5": 1,
                    "10": 17,
                    "20": 15,
                    "50": 0,
                    "100": 0,
                    "200": 0,
                    "500": 0
                },
                "CashDeposits": {},
                "Data": null
            }
        }
    ],
    "Expenses": [],
    "CashCorrections": [],
    "CurrencyID": "EUR",
    "CostPriceCurrencyID": "EUR",
    "FinancialPeriodAuditID": "EUW",
    "GenerationTime": "2021-06-12T18:00:29.3443732Z"
}
{
    "GenerationTime": "2021-05-07T08:55:06.1021798Z",
    "ID": "StockMutationID",
    "Product": {
        "ID": "ProductID",
        "Name": "ProductName",
        "BackendID": "ProductBackendID"
    },
    "OrganizationUnit": {
        "IsDeleted": false,
        "Description": null,
        "Notes": null,
        "ParentID": "OrganizationUnitParentID",
        "RegisterCashLimit": 1000.0000,
        "SafeCashLimit": null,
        "BackendRelationID": "OrganizationUnitBackendRelationID",
        "BackendCompanyID": "OrganizationUnitBackendCompanyID",
        "BranchNumber": "OrganizationUnitBranchNumber",
        "GlobalLocationNumber": null,
        "Address": {
            "AddressedTo": null,
            "Street": "OrganizationUnitStreet",
            "HouseNumber": "OrganizationUnitHouseNumber",
            "Address1": null,
            "Address2": null,
            "ZipCode": "OrganizationUnitZipCode",
            "Subdistrict": null,
            "District": null,
            "City": "OrganizationUnitCity",
            "State": null,
            "Region": null,
            "CountryID": "DE"
        },
        "Latitude": OrganizationUnitLatitude,
        "Longitude": OrganizationUnitLongitude,
        "Type": ["Shop", "Pickup", "EVA"],
        "Status": "Open",
        "Subnet": "OrganizationUnitSubnet",
        "BankAccount": null,
        "VatNumber": null,
        "RegistrationNumber": null,
        "EmailAddress": "OrganizationUnitMailAddress",
        "PhoneNumber": "OrganizationUnitPhoneNumber",
        "UseForAccounting": true,
        "IpAddress": "OrganizationUnitIpAddress",
        "CountryID": "DE",
        "LanguageID": "de",
        "CurrencyID": "EUR",
        "CostPriceCurrencyID": "EUR",
        "TimeZone": "Europe/Berlin",
        "Region": null,
        "CostPriceCalculationMethod": "Manual",
        "ID": "OrganizationUnitID",
        "Name": "OrganizationUnitName",
        "BackendID": "OrganizationUnitBackendID"
    },
    "SourceStockLabel": "Sellable",
    "DestinationStockLabel": "Sellable",
    "Quantity": 2,
    "UnitCost": 21.2500,
    "Reason": "FullStockCount",
    "Remark": "OptionalRemark",
    "OrderLineID": "OrderLineID",
    "BackendSystemID": null,
    "BackendID": null,
    "Ledgers": [{
            "StockLabel": "Sellable",
            "Quantity": 2
        }
    ]
}

File paths​

DataPath
Financial events/financialevents/Year/Month/Day/file-batch.json
Financial periods/financialperiods/FinancialPeriodID/Year/Month/Day/file.json
General ledgers/generalledger/Year/Month/Day/file-batch.json
Orders/orders/OrderNumber/Year/Month/Day/file.json
Invoices/invoices/InvoiceID/Year/Month/Day/file.json
Organization units/organizationunits/Year/Month/Day/file.json
Prices/prices/PriceListID/Year/Month/Day/file.json
Products/products/Year/Month/Day/file-batch.json
Stock/stock/OrganizationUnitID/Year/Month/Day/file.json
Stock mutations/stockmutations/StockMutationID/file.json
Event ledgers/eventledger/{Year}/{Month}/{Day}
Users/users/{Year}/{Month}/{Day}
Fulfilment results/fulfillmentresults/{orderID}/{year}/{month}/{day}/{guid}.json