All New WineDirect Data Lake

The All New WineDirect Data Lake provides records of all orders from Classic WineDirect and All New WineDirect.  It also provides Customer and WineClubs records from All New WineDirect. 

Data Refresh Frequency  

All New WineDirect Order, Customer, and Wine Club data are updated hourly, and WineDirect Classic Order data is updated daily. Folders for each day will contain orders that were CREATED OR MODIFIED on that date. An order can appear in multiple date folders if it gets updated on multiple dates. All dates and times are in UTC. All New WineDirect hourly data populates 5-10 minutes after the end of the hour.

Orders 

Data Structure 

Order Schema 

Files are in Parquet format.

Sample orders parquet file with 50 dummy orders record

Orders Sample File

Folder Structure 

Data is partitioned by Website/Tenant ID, Year, Month, Day, Source (wdc for Classic or anwd for All New WineDirect). Files can be downloaded for a specified year, month, or day. ANWD data is also available hourly starting Nov. 30, 2023

URL Structure 

For WineDirect websites on the "Classic" platform, use the Website ID folder to access order information only.

https://datalake.alpha.app.winedirect.com/website/{website_id}/orders/{year}/{month}/{day}

For new or migrated websites on ANWD, use the Tenant ID to access orders and all other objects (customers, products, etc.) listed below.

https://datalake.alpha.app.winedirect.com/tenant/{tenant_id}/orders/{year}/{month}/{day}

IMPORTANT NOTE: To access client data on the ANWD staging environment, use the Tenant ID and staging URL to access orders and all other objects (customers, products, etc.) listed below.

Hourly partitioned data is available in ANWD. Pass the hour to the URL to fetch the hourly data. A 5-10 minute lag exists before the previous hour's data is populated. 

https://datalake.alpha.app.winedirect.com/tenant/{tenant_id}/orders/{year}/{month}/{day}/{hour}

Examples 

• For Website ID abcd-123-XYZ-345, to get all orders from January 2023 use this URL (All capital alphabets for website ID)
  https://datalake.alpha.app.winedirect.com/website/ABCD-123-XYZ-345/orders/2023/1
  
• For Tenant ID abcd-123-XYZ-345, to get all orders from 2022 use this URL (All small alphabets for tenant ID)
  https://datalake.alpha.app.winedirect.com/tenant/abcd-123-xyz-345/orders/2022 
• For Tenant ID abcd-123-XYZ-345, to get all orders from 2023-11-09 hour 15 to 16 UTC use this URL (All small alphabets for tenant ID)
  https://datalake.alpha.app.winedirect.com/tenant/abcd-123-xyz-345/orders/2023/11/9/15
• For Website ID abcd-123-XYZ-345, to get all orders from 15^th January 2020 use this URL 
  https://datalake.alpha.app.winedirect.com/website/ABCD-123-XYZ-345/orders/2023/1/15  
  
  

IMPORTANT NOTE: Data from 1^st November 2022 onwards contains some duplicate records with a single order from Classic WineDirect being mapped to multiple orders in ANWD. We are actively working to resolve this issue and will provide an update when it has been resolved.

Steps to Download Records from Data Lake  

 1. Each Tenant or Website ID will have its own credentials. If you have an ANWD store, a user with StoreAdmin access can get the credentials from the Integrations section on the Admin Panel

2. Authenticate using the credentials <client_id> and <client_secret> provided

TextText
curl -X POST \
--user <client_id>:<client_secret> \
'https://datalake.alpha.app.winedirect.com/oauth2/token?grant_type=client_credentials' \
-H 'Content-Type: application/x-www-form-urlencoded'

3.  Authentication will return a token to use for accessing data. Use the token to fetch data from the Data Lake. The token will be valid for 60 min. 

Tenant

TextTextText
curl -X GET \
-H "Authorization: Bearer <TOKEN>" \
-H "Content-Type: application/json" https://datalake.alpha.app.winedirect.com/tenant/{tenant_id}/orders/{year}/{month}/{day} \
--output results.zip

Website

TextTextText
curl -X GET \
-H "Authorization: Bearer <TOKEN>" \
-H "Content-Type: application/json" https://datalake.alpha.app.winedirect.com/website/{website_id}/orders/{year}/{month}/{day} \
--output results.zip

Customers [Up to date as of Nov. 6, 2023] 

The Authentication process for Customer data is the same as the process for Orders data. The same token can be used if it is still valid. Customer data is only available for Tenant IDs and not Website IDs.

Below are the URLs available to download Customer related data:

Customers

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customers/<yyyy>/<mm>/<dd>
Customers Data Lake Schema 

Sample customers parquet file with 50 dummy customer records - Sample Customer Parquet File

Customer Tags

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customer_tags/<yyyy>/<mm>/<dd>
Customers Tags Data Lake Schema 

Customer Segments

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customer_segments/<yyyy>/<mm>/<dd>
Customer Segments Data Lake Schema 

Customer Date Type Options

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customer_date_type_options/<yyyy>/<mm>/<dd>
Customer Date Type Options Schema 

Customer Groups

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/customer_groups/<yyyy>/<mm>/<dd>
Customers Groups Data Lake Schema 

Wine Clubs [Up to date as of Nov. 6, 2023]

The authentication process for Wine Clubs data is the same as for Orders; the same token can be used if it is still valid. Wine Clubs data is only available for Tenant IDs and not Website IDs.

Below are the URLs available to download Wine Clubs related data:

Clubs

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/clubs/<yyyy>/<mm>/<dd>
Clubs Data Lake Schema 

Sample Clubs parquet file with 50 dummy clubs records - Sample Clubs Parquet File

Clubs Memberships

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/clubs_memberships/<yyyy>/<mm>/<dd>
Clubs Memberships Data Lake Schema 

Clubs Releases Products

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/clubs_release_products/<yyyy>/<mm>/<dd>
Clubs Releases Products Data Lake Schema 

Clubs Releases

https://datalake.alpha.app.winedirect.com/tenant/<tenantid>/clubs_releases/<yyyy>/<mm>/<dd>
Clubs Releases Data Lake Schema