All New WineDirect Data Lake
  • 08 Jan 2024
  • 3 Minutes to read
  • Dark
    Light
  • PDF

All New WineDirect Data Lake

  • Dark
    Light
  • PDF

Article Summary

The All New WineDirect Data Lake provides records of all orders from Classic WineDirect and All New WineDirect.  It also provides Customer and WineClubs records from All New WineDirect. 


Orders 

Data Structure 

Order Schema 

Files are in Parquet format.

Sample orders parquet file with 50 dummy orders record

Orders Sample File


Folder Structure 

Data is partitioned by Website/Tenant ID, Year, Month, Day, Source (wdc for Classic or anwd for All New WineDirect). Files can be downloaded for a specified year, month, or day. ANWD data is also available hourly starting Nov. 30, 2023

URL Structure 

For WineDirect websites on the "Classic" platform, use the Website ID folder to access order information only.

https://datalake.alpha.app.winedirect.com/website/{website_id}/orders/{year}/{month}/{day}

For new or migrated websites on ANWD, use the Tenant ID to access orders and all other objects (customers, products, etc.) listed below.

https://datalake.alpha.app.winedirect.com/tenant/{tenant_id}/orders/{year}/{month}/{day}

IMPORTANT NOTE: To access client data on the ANWD staging environment, use the Tenant ID and staging URL to access orders and all other objects (customers, products, etc.) listed below.

Hourly partitioned data is available in ANWD. Pass the hour to the URL to fetch the hourly data. A 5-10 minute lag exists before the previous hour's data is populated. 

https://datalake.alpha.app.winedirect.com/tenant/{tenant_id}/orders/{year}/{month}/{day}/{hour}

Examples 

  • For Website ID abcd-123-XYZ-345, to get all orders from January 2023 use this URL (All capital alphabets for website ID)
    https://datalake.alpha.app.winedirect.com/website/ABCD-123-XYZ-345/orders/2023/1
  • For Tenant ID abcd-123-XYZ-345, to get all orders from 2022 use this URL (All small alphabets for tenant ID)
    https://datalake.alpha.app.winedirect.com/tenant/abcd-123-xyz-345/orders/2022 
  • For Tenant ID abcd-123-XYZ-345, to get all orders from 2023-11-09 hour 15 to 16 UTC use this URL (All small alphabets for tenant ID)
    https://datalake.alpha.app.winedirect.com/tenant/abcd-123-xyz-345/orders/2023/11/9/15
  • For Website ID abcd-123-XYZ-345, to get all orders from 15th January 2020 use this URL 
    https://datalake.alpha.app.winedirect.com/website/ABCD-123-XYZ-345/orders/2023/1/15  

IMPORTANT NOTE: Data from 1st November 2022 onwards contains some duplicate records with a single order from Classic WineDirect being mapped to multiple orders in ANWD. We are actively working to resolve this issue and will provide an update when it has been resolved.

Data Refresh Frequency  

Data is updated daily. Folders for each day will contain orders that were CREATED OR MODIFIED on that date. An order can appear in multiple date folders if it gets updated on multiple dates. All dates and times are in UTC.ANWD Orders Hourly data populates 5-10 min after the end of the hour.

Steps to Download Records from Data Lake  

 1. Each Tenant or Website ID will have its own credentials. If you have an ANWD store, a user with StoreAdmin access can get the credentials from the Integrations section on the Admin Panel


2. Authenticate using the credentials <client_id> and <client_secret> provided

Text
curl -X POST \ --user <client_id>:<client_secret> \ 'https://datalake.alpha.app.winedirect.com/oauth2/token?grant_type=client_credentials' \ -H 'Content-Type: application/x-www-form-urlencoded'

3.  Authentication will return a token to use for accessing data. Use the token to fetch data from the Data Lake. The token will be valid for 60 min. 

Tenant

TextText
curl -X GET \ -H "Authorization: Bearer <TOKEN>" \ -H "Content-Type: application/json" https://datalake.alpha.app.winedirect.com/tenant/{tenant_id}/orders/{year}/{month}/{day} \ --output results.zip

Website

TextText
curl -X GET \ -H "Authorization: Bearer <TOKEN>" \ -H "Content-Type: application/json" https://datalake.alpha.app.winedirect.com/website/{website_id}/orders/{year}/{month}/{day} \ --output results.zip

Customers [Up to date as of Nov. 6, 2023] 

The Authentication process for Customer data is the same as the process for Orders data. The same token can be used if it is still valid. Customer data is only available for Tenant IDs and not Website IDs.

Below are the URLs available to download Customer related data:

Customers

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customers/<yyyy>/<mm>/<dd>
 

Sample customers parquet file with 50 dummy customer records - Sample Customer Parquet File

Customer Tags

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customer_tags/<yyyy>/<mm>/<dd>
 

Customer Segments

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customer_segments/<yyyy>/<mm>/<dd>
 

Customer Date Type Options

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customer_date_type_options/<yyyy>/<mm>/<dd>
 

Customer Groups

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customer_groups/<yyyy>/<mm>/<dd>
 

Wine Clubs [Up to date as of Nov. 6, 2023]

The authentication process for Wine Clubs data is the same as for Orders; the same token can be used if it is still valid. Wine Clubs data is only available for Tenant IDs and not Website IDs.

Below are the URLs available to download Wine Clubs related data:

Clubs

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/clubs/<yyyy>/<mm>/<dd>
 

Sample Clubs parquet file with 50 dummy clubs records - Sample Clubs Parquet File

Clubs Memberships

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/clubs_memberships/<yyyy>/<mm>/<dd>
 

Clubs Releases Products

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/clubs_release_products/<yyyy>/<mm>/<dd>
 

Clubs Releases

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/clubs_releases/<yyyy>/<mm>/<dd>
 


Was this article helpful?