Skip to main content

Data lake exports

EVA is an event-driven platform. Everything from transactions and returns to stock movements and mutations is logged and registered as an event in real-time. Using the data from these events you can create your own custom dashboards and reports.

Pushing data to a data lake

Datalake Provider Permissions

Ensure that your DataLake setup has read, write, and delete permissions on the provider's side to enable seamless data export. Without these permissions, DataLake cannot perform necessary exports, resulting in failures.

Note that required permissions can vary among DataLake providers; verify the specific needs with your provider

The most flexible and future-proof way of creating dashboard and reports based on your data, is using a data lake. A data lake is a single store of enterprise data including raw copies of source system data. This setup has multiple benefits over a traditional database or data warehouse:

Scalable Unlike a traditional data warehouse, data lakes offer scalability, typically at lower costs.

Versatile A data lake can store both structured and unstructured data from diverse sources. In other words, it can store XML, logs, multimedia, sensor data, chat, social data, binary, and people data.

Schema Flexibility For traditional schema, you need to have your data in a structured format. Traditional data warehouse products are schema based. But for analytics, this could prove to be a glitch as the data needs to be analyzed in its raw form.

Setting up a data lake, including vendor selection and contracting, is a client responsibility. Of course, New Black can assist in this process. We recommend using a Standard general-purpose v2 Azure storage account.

Setting up your data lake

There are several kinds of data lake integrations possible. These are all described, with their own specific settings, here below. In order to enable the settings, we do need to enable a plugin behind the scenes. Please reach out to your Strategic Lead.

Azure

To set up your data lake, make sure you have a regular V2 Azure storage account. EVA needs your account's information to export your data. The following settings are absolutely necessary:

SettingDescription
Azure:DataLake:AccountNameAzure Account name
Azure:DataLake:AccountKeyAzure Account Key
Azure:DataLake:FileSystemThe "container" name in the storage account
Azure:DataLake:FolderDesired folder structure, for example: RAW/EVA will create a RAW folder with an EVA subfolder
Azure:DataLake:ServiceUrlYour Azure Service URL

The following settings are optional:

SettingDescription
Azure:DataLake:TenantTenant ID for Azure AD directory authentication.
Azure:DataLake:ClientIDApplication ID for identifying the Azure AD app.
Azure:DataLake:ClientSecretClient Secret for authenticating the Azure AD app.

AWS

The following settings apply if you make use of AWS data lake. These settings make use of the same cron jobs to trigger the exports (see Config below), regardless of the Azure name.

SettingDescription
AWS:DataLake:RegionThe region where the S3 bucket lives.
AWS:DataLake:S3BucketThe name of the S3 bucket to store the files in.
AWS:DataLake:AccessKey-
AWS:DataLake:SecretKey-
AWS:DataLake:Folder(optional) The folder to store the files in the bucket. If not set, we use the root folder.
AWS:DataLake:EdgeServiceUrl(optional) This setting can include your Edge service URL, which can be provided by AWS upon request. This is a URL which allows for higher throughput by lowering latency.

Google BigQuery

The following settings apply if you make use of Google BigQuery data lake. These settings make use of the same cron jobs to trigger the exports (see Config below), regardless of the Azure name.

SettingDescription
GCP:DataLake:BucketThe name of the bucket to store the files in.
GCP:DataLake:CredentialThe credential in JSON format.
GCP:DataLake:Folder(optional) The folder to store the files in the bucket. If not set, we use the root folder.

Data Pack

EVA exports the following data to your data lake:

  1. Orders
  2. Financial periods
  3. Financial events
  4. *Products
  5. *Replenishment products
  6. *Stock
  7. *Stock mutations
  8. *Organization units
  9. Organization unit sets
  10. General ledgers
  11. *Prices
  12. Sales invoices
  13. Users
  14. Fulfilment results - this is a very heavy export. Either manually enable it with DataLake:ExportFulfillmentResults or rely on Admin Suite's Fulfilment chapter to view the results.
  15. Shipments
  16. *User-OU relations
  17. User tasks
  18. *User cards
  19. Event ledgers
  20. Discounts
  21. Discount campaigns
  22. Discount layers
  23. Wishlists
  24. Assortments - enable it manually by setting DataLake:ExportAssortments to true.
  25. Repairs
  26. User-bought products
  27. Cases
  28. ZonedCycleCounts
  29. UserLoyaltyTransaction
  30. Payment transactions

Each data pack preceded by a * is activated by means of a scheduled task (see Config), which means their frequency is configurable. All others will stream towards your data lake based on changes.

Reporting and data

For any sort of reporting or reconciliation we recommend that you use data lake exports, and not APIs. Furthermore, if there's data you wish to include that are not mentioned in the above list, please raise this through a sprint.

Event ledgers

Aside from the above list of events, EVA also facilitates the export of event ledgers. Because you might not want your DataLake to be stuffed with event ledgers, you'll need to explicitly enable this by switching DataLake:ExportEventLedger to true.

Event ledger types
- AuthenticationAttempt
- Login
- Logout
- OrderCreated
- OrderLinesPlaced
- OrderAutoCompleted
- ManualReturnOrderCreated
- PaymentCreated
- PaymentFinalized
- CashDrawerOpened
- CashTransaction
- CashDrawerClosed
- CashDrawerClosedTimeout
- CashDrawerClosingDeviation
- CashDrawerOpeningDeviation
- UserTaskStarted
- UserTaskCompleted
- UserDetachedFromUserTask
- UserTaskIgnored
- DiscountUsed
- ManualDiscountGiven
- InvoiceCreated
- InvoicePrinted
- InvoiceDuplicatePrinted
- InvoiceMailed
- InvoiceDownloaded
- InvoiceArchived
- ReceiptPrinted
- ReceiptDuplicatePrinted
- ReceiptPrintFailed
- ReceiptReprinted
- ReceiptReprintFailed
- MessageTemplateUpdated
- SettingUpdated
- EVAInstanceStarted
- FinancialPeriodClosed
- FinancialReportingPeriodClosed
- FinancialPeriodAuditExported
- FiscalPeriodArchived
- CashExpense
- CashDeposit
- CashCorrection
- CashMovement
- UserAttachedToRole
- UserRemovedFromRole
- FunctionalityAddedToUser
- FunctionalityRemovedFromUser
- FunctionalityAddedToRole
- FunctionalityRemovedFromRole
- OrganizationUnitChanged
- OrganizationUnitCreated
- XReport
- ZReport
- FincialPeriodClosingImpedimentIgnored

Config

Most data packs are exported by default; you can enable the other exports by setting up the following cron tasks:

  • EVA.DataLake.Tasks.ExportPriceLists
  • EVA.DataLake.Tasks.ExportProducts
  • EVA.DataLake.Tasks.ExportOrganizationUnits
  • EVA.DataLake.Tasks.ExportReplenishmentProducts
  • EVA.DataLake.Tasks.ExportStock
    • Can also be used to configure one-off exports based on a specific DateTime.
  • EVA.DataLake.Tasks.ExportStockMutations
  • EVA.DataLake.Tasks.ExportUserStoreRelations
  • EVA.DataLake.Tasks.ExportUsercards
  • EVA.DataLake.Tasks.ExportAssortments

These are all full exports and are not necessarily exported often. You may want to set these to run at midnight for example. An exception to this however is EVA.DataLake.Tasks.ExportStockMutations, which you may want to run every 10 or 15 minutes for example.

Examples

Take into account that the following lists of examples are just that: examples. They give a realistic view of what a data export in a fully configured environment might look like, but they can be outdated and may be missing data which you currently are getting from your exports. For a comprehensive view of all exportable properties, see the Overview of all exportable properties below.

{
"ID": "FinancialPeriodID",
"FinancialPeriodNumber": null,
"PreviousFinancialPeriodID": "PreviousFinancialPeriodID",
"OpeningTime": "2021-06-12T07:19:47.963",
"ClosingTime": null,
"AutoClosed": false,
"Status": "Open",
"OrganizationUnitID": "OUID",
"User": {
"ID": "UserID",
"Nickname": "UserNickname",
"EmailAddress": "UserMailAddress",
"FirstName": "UserFirstName",
"LastName": "UserLastName",
"Gender": null,
"LanguageID": "fr",
"CountryID": "FR",
"Salutation": null
},
"CashDeposits": [],
"CashJournals": [{
"FinancialPeriodID": "FinancialPeriodID",
"PreviousCashJournalID": "PreviousFinancialPeriodID",
"PaymentTypeID": "PaymentTypeID",
"PaymentType": {
"PaymentMethodID": "PaymentMethodID",
"Name": "Cash",
"Code": "CASH",
"LedgerClassID": null,
"BackendRelationID": null,
"Category": "Cash",
"CashJournalMethod": "OpenAndClose"
},
"CurrencyID": "EUR",
"Type": "Default",
"OpeningTime": "2021-06-12T07:28:07.453",
"ClosingTime": "2021-06-12T17:29:54.487",
"OpeningAmount": 491.8800,
"ClosingAmount": 490.8800,
"OpeningDeviation": 0.0000,
"ClosingDeviation": null,
"StationID": "StationID",
"DeviceID": "DeviceID",
"OpeningDetailsObject": {
"Coins": {
"1": 2,
"2": 7,
"0.01": 3,
"0.02": 0,
"0.05": 3,
"0.1": 7,
"0.2": 0,
"0.5": 0
},
"BankNotes": {
"5": 1,
"10": 17,
"20": 15,
"50": 0,
"100": 0,
"200": 0,
"500": 0
},
"CashDeposits": {}
},
"ClosingDetailsObject": {
"Coins": {
"1": 1,
"2": 7,
"0.01": 3,
"0.02": 0,
"0.05": 3,
"0.1": 7,
"0.2": 0,
"0.5": 0
},
"BankNotes": {
"5": 1,
"10": 17,
"20": 15,
"50": 0,
"100": 0,
"200": 0,
"500": 0
},
"CashDeposits": {},
"Data": null
}
}
],
"Expenses": [],
"CashCorrections": [],
"CurrencyID": "EUR",
"CostPriceCurrencyID": "EUR",
"FinancialPeriodAuditID": "EUW",
"GenerationTime": "2021-06-12T18:00:29.3443732Z"
}
{
"GenerationTime": "2021-05-07T08:55:06.1021798Z",
"ID": "StockMutationID",
"Product": {
"ID": "ProductID",
"Name": "ProductName",
"BackendID": "ProductBackendID"
},
"OrganizationUnit": {
"IsDeleted": false,
"Description": null,
"Notes": null,
"ParentID": "OrganizationUnitParentID",
"RegisterCashLimit": 1000.0000,
"SafeCashLimit": null,
"BackendRelationID": "OrganizationUnitBackendRelationID",
"BackendCompanyID": "OrganizationUnitBackendCompanyID",
"BranchNumber": "OrganizationUnitBranchNumber",
"GlobalLocationNumber": null,
"Address": {
"AddressedTo": null,
"Street": "OrganizationUnitStreet",
"HouseNumber": "OrganizationUnitHouseNumber",
"Address1": null,
"Address2": null,
"ZipCode": "OrganizationUnitZipCode",
"Subdistrict": null,
"District": null,
"City": "OrganizationUnitCity",
"State": null,
"Region": null,
"CountryID": "DE"
},
"Latitude": OrganizationUnitLatitude,
"Longitude": OrganizationUnitLongitude,
"Type": ["Shop", "Pickup", "EVA"],
"Status": "Open",
"Subnet": "OrganizationUnitSubnet",
"BankAccount": null,
"VatNumber": null,
"RegistrationNumber": null,
"EmailAddress": "OrganizationUnitMailAddress",
"PhoneNumber": "OrganizationUnitPhoneNumber",
"UseForAccounting": true,
"IpAddress": "OrganizationUnitIpAddress",
"CountryID": "DE",
"LanguageID": "de",
"CurrencyID": "EUR",
"CostPriceCurrencyID": "EUR",
"TimeZone": "Europe/Berlin",
"Region": null,
"CostPriceCalculationMethod": "Manual",
"ID": "OrganizationUnitID",
"Name": "OrganizationUnitName",
"BackendID": "OrganizationUnitBackendID"
},
"SourceStockLabel": "Sellable",
"DestinationStockLabel": "Sellable",
"Quantity": 2,
"UnitCost": 21.2500,
"Reason": "FullStockCount",
"Remark": "OptionalRemark",
"OrderLineID": "OrderLineID",
"BackendSystemID": null,
"BackendID": null,
"Ledgers": [{
"StockLabel": "Sellable",
"Quantity": 2
}
]
}

File paths

DataPath
Financial events/financialevents/Year/Month/Day/file-batch.json
Financial periods/financialperiods/FinancialPeriodID/Year/Month/Day/file.json
General ledgers/generalledger/Year/Month/Day/file-batch.json
Orders/orders/OrderNumber/Year/Month/Day/file.json
Invoices/invoices/InvoiceID/Year/Month/Day/file.json
Organization units/organizationunits/Year/Month/Day/file.json
Organization unit sets/organizationunitsets/Year/Month/Day/file.json
Prices/prices/PriceListID/Year/Month/Day/file.json
Products/products/Year/Month/Day/file-batch.json
Stock/stock/OrganizationUnitID/Year/Month/Day/file.json
Stock mutations/stockmutations/StockMutationID/file.json
Event ledgers/eventledger/Year/Month/Day
Users/users/{year}/{month}/{day}/{id}/{uid}.json
User-OU relations/users/{year}/{month}/{day}/{uid}.json
Fulfilment results/fulfillmentresults/orderID/year/month/day/guid.json
User cards/usercards/Year/Month/Day/guid.json
Discounts/discounts/Year/Month/Day
Discount campaigns/discountcampaigns/Year/Month/Day
Discount layers/discountlayers/Year/Month/Day
Wishlists/wishlists/Year/Month/Day
Assortments/assortments/{id}/{year}/{month}/{day}/{uid}.json
User-bought products/userboughtproducts/year/month/day/id
Cases/cases/Year/Month/Day
ZonedCycleCounts/zonedcyclecounts/Year/Month/Day
Replenishment products/replenishment/ID/Year/Month/Day
User loyalty transactions/UserLoyaltyTransaction/ID/Year/Month/Day/ID
Payment transactions/paymenttransactions/{year}/{month}/{day}/{id}/{uid}.json

Overview of all exportable properties

We've shown some real-life examples of our data lake objects out of our own environment, but if you really want to dive into it, the following lists show every single possible property for each export. This way you can ascertain what properties you can expect and map to.

{
"ID": "string",
"Type": "string",
"Lines": [
{
"ID": "string",
"Type": "string",
"Remark": "string",
"SoldBy": {
"ID": "string",
"Gender": "string",
"LastName": "string",
"Nickname": "string",
"BackendID": "string",
"CountryID": "string",
"FirstName": "string",
"LanguageID": "string",
"Salutation": "string",
"DateOfBirth": "string",
"PhoneNumber": "string",
"CustomFields": [
{
"Name": "string",
"BackendID": "string",
"CustomFieldID": 123
}
],
"EmailAddress": "string",
"PlaceOfBirth": "string",
"BackendSystemID": "string"
},
"Amounts": {
"TotalAmount": 1.23,
"DiscountAmount": 1.23,
"NetTotalAmount": 1.23,
"TotalTaxAmount": 1.23,
"TotalAmountInTax": 1.23,
"NetTotalTaxAmount": 1.23,
"NetTotalAmountInTax": 1.23
},
"Product": {
"ID": "string",
"Name": "string",
"BackendID": "string",
"LedgerClassID": "string",
"ManufacturerID": "string",
"BackendSystemID": "string"
},
"TaxRate": 1.23,
"ParentID": "string",
"UnitCost": 1.23,
"Discounts": [
{
"Amount": 1.23,
"DiscountID": "string",
"CompliancyRule": "string",
"DiscountBackendID": "string",
"DiscountOrderLineID": "string"
}
],
"IsOrdered": false,
"IsShipped": false,
"UnitPrice": 1.23,
"CurrencyID": "string",
"IsCarryOut": false,
"IsDelivery": false,
"IsExported": false,
"IsInvoiced": false,
"TaxDetails": [
{
"Name": "string",
"Rate": 1.23,
"Type": "string",
"Amount": 1.23
}
],
"Description": "string",
"CreationTime": "string",
"CustomFields": [
{
"Name": "string",
"BackendID": "string",
"CustomFieldID": 123
}
],
"ReturnReason": {
"ID": "string",
"Name": "string",
"BackendID": "string"
},
"SerialNumber": "string",
"IsReservation": false,
"RequestedDate": "string",
"DiscountAmount": 1.23,
"ShippingMethod": "string",
"QuantityOrdered": 123,
"QuantityShipped": 123,
"ReturnedOrderID": "string",
"DiscountCouponID": "string",
"QuantityExported": 123,
"QuantityReserved": 123,
"OriginalUnitPrice": 1.23,
"QuantityCancelled": 123,
"QuantityCommitted": 123,
"QuantityDelivered": 123,
"AvailabilityStatus": "string",
"DiscountCouponCode": "string",
"CostPriceCurrencyID": "string",
"ProductRequirements": [
{
"ID": "string",
"Name": "string"
}
],
"ReturnedOrderLineID": "string",
"TotalQuantityToShip": 123,
"LastModificationTime": "string",
"UnitPricePriceListID": "string",
"ProductVariationLines": [
{
"ID": "string",
"Label": "string",
"Value": "string"
}
],
"OriginalUnitPriceInTax": 1.23,
"ExpectedAvailabilityDate": "string",
"UnitPriceBeforeCorrection": 1.23,
"UnitPriceCorrectionReason": {
"ID": "string",
"Name": "string",
"Description": "string"
}
}
],
"IsPaid": false,
"Remark": "string",
"Amounts": {
"Open": {
"InTax": 1.23,
"Capturable": 1.23,
"PendingInTax": 1.23
},
"Paid": {
"Amount": 1.23,
"Pending": 1.23,
"Captured": 1.23
},
"Taxes": [
{
"Base": 1.23,
"Rate": 1.23,
"Total": 1.23,
"Amount": 1.23
}
],
"Total": {
"Tax": 1.23,
"InTax": 1.23,
"Amount": 1.23
},
"Discounts": {
"PerDiscount": [
{
"Total": 1.23,
"Discount": {
"Description": "string"
},
"DiscountID": 123
}
],
"TotalDiscountAmount": 1.23
},
"CurrencyID": "string",
"UnroundedTotal": {
"Tax": 1.23,
"InTax": 1.23,
"Amount": 1.23
},
"OpenAmountIsZero": false
},
"GroupID": "string",
"Customer": {
"ID": "string",
"Gender": "string",
"LastName": "string",
"Nickname": "string",
"BackendID": "string",
"CountryID": "string",
"FirstName": "string",
"LanguageID": "string",
"Salutation": "string",
"DateOfBirth": "string",
"PhoneNumber": "string",
"CustomFields": [
{
"Name": "string",
"BackendID": "string",
"CustomFieldID": 123
}
],
"EmailAddress": "string",
"PlaceOfBirth": "string",
"BackendSystemID": "string"
},
"Payments": [
{
"ID": "string",
"Code": "string",
"Amount": 1.23,
"DeviceID": "string",
"IsRefund": false,
"BackendID": "string",
"CurrencyID": "string",
"PaidAmount": 1.23,
"Description": "string",
"IsConfirmed": false,
"PaymentDate": "string",
"CustomFields": [
{
"Name": "string",
"BackendID": "string",
"CustomFieldID": 123
}
],
"PaymentMethod": "string",
"BackendSystemID": "string",
"BackendRelationID": "string"
}
],
"BackendID": "string",
"IsShipped": false,
"UserAgent": "string",
"CurrencyID": "string",
"Properties": 123,
"Description": "string",
"InvoiceDate": "string",
"IsCompleted": false,
"PaymentDate": "string",
"CreationTime": "string",
"CustomFields": [
{
"Name": "string",
"BackendID": "string",
"CustomFieldID": 123
}
],
"CustomStatus": [
"string"
],
"GiftWrapping": {
"Wraps": [
{
"Lines": [
{
"Quantity": 123,
"OrderLineID": "string"
}
],
"Message": "string",
"GreetingCardProductID": "string"
}
],
"Message": "string",
"WrapOrder": false,
"WrapIndividually": false,
"GreetingCardProductID": "string"
},
"OrderLedgers": [
{
"ID": 123,
"Type": "string",
"NewValue": "string",
"OldValue": "string",
"CreatedBy": {
"ID": "string",
"Gender": "string",
"LastName": "string",
"Nickname": "string",
"BackendID": "string",
"CountryID": "string",
"FirstName": "string",
"LanguageID": "string",
"Salutation": "string",
"DateOfBirth": "string",
"PhoneNumber": "string",
"CustomFields": [
{
"Name": "string",
"BackendID": "string",
"CustomFieldID": 123
}
],
"EmailAddress": "string",
"PlaceOfBirth": "string",
"BackendSystemID": "string"
},
"Description": "string",
"OrderLineID": "string",
"CreationTime": "string"
}
],
"ShipmentDate": "string",
"PlacementDate": "string",
"BillingAddress": {
"City": "string",
"State": "string",
"Region": "string",
"Street": "string",
"ZipCode": "string",
"Address1": "string",
"Address2": "string",
"District": "string",
"LastName": "string",
"CountryID": "string",
"FirstName": "string",
"AddressedTo": "string",
"HouseNumber": "string",
"Subdistrict": "string"
},
"CompletionDate": "string",
"GenerationTime": "string",
"BackendSystemID": "string",
"OrderCustomType": {
"ID": "string",
"Name": "string",
"BackendID": "string",
"Description": "string"
},
"ShippingAddress": {
"City": "string",
"State": "string",
"Region": "string",
"Street": "string",
"ZipCode": "string",
"Address1": "string",
"Address2": "string",
"District": "string",
"LastName": "string",
"CountryID": "string",
"FirstName": "string",
"AddressedTo": "string",
"HouseNumber": "string",
"Subdistrict": "string"
},
"OrderCustomStatus": {
"ID": "string",
"Name": "string",
"BackendID": "string",
"Description": "string"
},
"LastModificationTime": "string",
"PickupOrganizationUnit": {
"ID": "string",
"Name": "string",
"BackendID": "string"
},
"ShipToOrganizationUnit": {
"ID": "string",
"Name": "string",
"BackendID": "string"
},
"SoldToOrganizationUnit": {
"ID": "string",
"Name": "string",
"BackendID": "string"
},
"ShipFromOrganizationUnit": {
"ID": "string",
"Name": "string",
"BackendID": "string"
},
"SoldFromOrganizationUnit": {
"ID": "string",
"Name": "string",
"BackendID": "string"
},
"TotalAmountIncludingTaxes": 1.23,
"OriginatingOrganizationUnit": {
"ID": "string",
"Name": "string",
"BackendID": "string"
}
}

Manual re-exporting

By using the DataLakeBackfill service you can manually re-export specific Data lake documents.

Authorization

You need a specific (unscoped) functionality on your role called DataLakeBackfill in order to be able to use the datalake backfill service(s).

These types are currently supported:

  • FinancialPeriods = 1
  • Orders = 2
  • Invoices = 3
  • Users = 4
  • Discounts = 5 (this will trigger all connected data to the discount)
  • UserBatches = 6 (can only be used with From and To)
  • Cases = 7
  • Shipments = 8
  • ZonedCycleCounts = 9
  • UserTasks = 10
  • StockMutations = 11
  • UserLoyaltyPointMutations = 12 (these need to be the user IDs of which the loyalty point mutation will be exported)
  • LoyaltyProgramGroups = 13 (
  • LoyaltyPrograms = 14

Specify either a single ID, a list of IDs, or a date/time range From and To (inclusive).

As mandatory safety feature, specify the MaxNumberOfDocuments you'd like to run this for. When the amount of documents through above selection exceeds this number, this service will export NOTHING and return to you an error with the amount of documents it prevented from exporting, so you can adjust your selection or expand your max.

danger

Triggering large amounts of data may impact system performance!

Once done, the ID returned in the response can be used for subsequent services DataLakeBackfillStatus and DataLakeBackfillCancel.