Skip to content

Commit 8097112

Browse files
soispokesoispoke
authored
T 13697 nft.trades/mints/burns/transactions/fees rework #1222 (#1242)
Brief comments on the purpose of your changes: This (very large) PR is the first PR that implements the new NFT spell design on Dune V2. Changes affect: - Opensea trades/transactions/mints/burns/fees on Ethereum and Solana - MagicEden trades/transactions/mints on Solana New columns were added: blockchain, project, version, block_time, token_id, collection, amount_usd, token_standard, trade_type, number_of_items, trade_category, evt_type, seller, buyer, amount_original, amount_raw, currency_symbol, currency_contract, nft_contract_address, project_contract_address, aggregator_name, aggregator_address, tx_hash, block_number, platform_fee_amount_raw, platform_fee_amount, platform_fee_amount_usd, platform_fee_percentage, royalty_fee_amount_raw, royalty_fee_amount, royalty_fee_amount_usd, royalty_fee_percentage, royalty_fee_receive_address, royalty_fee_currency_symbol, block_number, tx_from, tx_to, unique_trade_id Made sure tests work w/ the new changes, run time is acceptable, fields match between Solana and Ethereum spells (although some fields are missing in Solana spells). I also implemented a temporary fix the bytea2numeric issue on Dune v2 that affects OpenSea ethereum spells: Token IDs larger than 64 bits now have the following mention: 'Token ID is larger than 64 bits and can not be displayed' until we find a better solution *For Dune Engine V2* I've checked that: * [x] I tested the query on dune.com after compiling the model with dbt compile (compiled queries are written to the target directory) * [x] I used "refs" to reference other models in this repo and "sources" to reference raw or decoded tables * [x] if adding a new model, I added a test * [x] the filename is unique and ends with .sql * [x] each sql file is a select statement and has only one view, table or function defined * [x] column names are `lowercase_snake_cased` When you are ready for a review, tag duneanalytics/data-experience. We will re-open your forked pull request as an internal pull request. Then your spells will run in dbt and the logs will be avaiable in Github Actions DBT Slim CI. This job will only run the models and tests changed by your PR compared to the production project.
1 parent b711488 commit 8097112

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+2212
-503
lines changed

‎spellbook/dbt_project.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@ seed-paths: ["seeds"]
1616
macro-paths: ["macros"]
1717
snapshot-paths: ["snapshots"]
1818

19-
2019
target-path: "target" # directory which will store compiled SQL files
2120
clean-targets: # directories to be removed by `dbt clean`
2221
- "target"
@@ -31,7 +30,9 @@ models:
3130
+materialized: view
3231
nft:
3332
+schema: nft
34-
+materialized: view
33+
ethereum:
34+
+schema: nft_ethereum
35+
+materialized: view
3536
uniswap:
3637
+schema: uniswap
3738
+materialized: view
@@ -82,7 +83,6 @@ models:
8283
+schema: transfers_ethereum
8384
+materialized: view
8485

85-
8686
seeds:
8787
spellbook:
8888
balances:

‎spellbook/macros/alter_table_properties.sql‎

Lines changed: 97 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -82,58 +82,122 @@ ALTER VIEW balances_ethereum.erc1155_latest SET TBLPROPERTIES('dune.public'='tru
8282
'dune.data_explorer.contributors'='["soispoke"]');
8383
{% endset %}
8484

85-
{% set magiceden_trades %}
86-
ALTER TABLE magiceden.trades SET TBLPROPERTIES('dune.public'='true',
87-
'dune.data_explorer.blockchains'='["solana"]',
85+
{% set opensea_transactions %}
86+
ALTER TABLE opensea.transactions SET TBLPROPERTIES('dune.public'='true',
87+
'dune.data_explorer.blockchains'='["ethereum","solana"]',
8888
'dune.data_explorer.category'='abstraction',
8989
'dune.data_explorer.abstraction.type'='project',
90-
'dune.data_explorer.abstraction.name'='magiceden',
90+
'dune.data_explorer.abstraction.name'='opensea',
9191
'dune.data_explorer.contributors'='["soispoke"]');
9292
{% endset %}
9393

94-
{% set nft_trades %}
95-
ALTER TABLE nft.trades SET TBLPROPERTIES('dune.public'='true',
94+
{% set opensea_trades %}
95+
ALTER VIEW opensea.trades SET TBLPROPERTIES('dune.public'='true',
9696
'dune.data_explorer.blockchains'='["ethereum","solana"]',
9797
'dune.data_explorer.category'='abstraction',
98-
'dune.data_explorer.abstraction.type'='sector',
99-
'dune.data_explorer.abstraction.name'='nft',
98+
'dune.data_explorer.abstraction.type'='project',
99+
'dune.data_explorer.abstraction.name'='opensea',
100100
'dune.data_explorer.contributors'='["soispoke"]');
101101
{% endset %}
102102

103-
{% set opensea_active_traders_day %}
104-
ALTER VIEW opensea.active_traders_day SET TBLPROPERTIES('dune.public'='true',
103+
{% set opensea_mints %}
104+
ALTER VIEW opensea.mints SET TBLPROPERTIES('dune.public'='true',
105105
'dune.data_explorer.blockchains'='["ethereum","solana"]',
106106
'dune.data_explorer.category'='abstraction',
107107
'dune.data_explorer.abstraction.type'='project',
108108
'dune.data_explorer.abstraction.name'='opensea',
109-
'dune.data_explorer.contributors'='["rchen8"]');
109+
'dune.data_explorer.contributors'='["soispoke"]');
110110
{% endset %}
111111

112-
{% set opensea_trades %}
113-
ALTER TABLE opensea.trades SET TBLPROPERTIES('dune.public'='true',
112+
{% set opensea_burns %}
113+
ALTER VIEW opensea.burns SET TBLPROPERTIES('dune.public'='true',
114114
'dune.data_explorer.blockchains'='["ethereum","solana"]',
115115
'dune.data_explorer.category'='abstraction',
116116
'dune.data_explorer.abstraction.type'='project',
117117
'dune.data_explorer.abstraction.name'='opensea',
118-
'dune.data_explorer.contributors'='["rchen8","soispoke"]');
118+
'dune.data_explorer.contributors'='["soispoke"]');
119119
{% endset %}
120120

121-
{% set opensea_txns_day %}
122-
ALTER VIEW opensea.txns_day SET TBLPROPERTIES('dune.public'='true',
121+
{% set opensea_fees %}
122+
ALTER VIEW opensea.fees SET TBLPROPERTIES('dune.public'='true',
123123
'dune.data_explorer.blockchains'='["ethereum","solana"]',
124124
'dune.data_explorer.category'='abstraction',
125125
'dune.data_explorer.abstraction.type'='project',
126126
'dune.data_explorer.abstraction.name'='opensea',
127-
'dune.data_explorer.contributors'='["rchen8"]');
127+
'dune.data_explorer.contributors'='["soispoke"]');
128128
{% endset %}
129129

130-
{% set opensea_volume_day %}
131-
ALTER VIEW opensea.volume_day SET TBLPROPERTIES('dune.public'='true',
132-
'dune.data_explorer.blockchains'='["ethereum","solana"]',
130+
131+
{% set magiceden_transactions %}
132+
ALTER TABLE magiceden.transactions SET TBLPROPERTIES('dune.public'='true',
133+
'dune.data_explorer.blockchains'='["solana"]',
133134
'dune.data_explorer.category'='abstraction',
134135
'dune.data_explorer.abstraction.type'='project',
135-
'dune.data_explorer.abstraction.name'='opensea',
136-
'dune.data_explorer.contributors'='["rchen8"]');
136+
'dune.data_explorer.abstraction.name'='magiceden',
137+
'dune.data_explorer.contributors'='["soispoke"]');
138+
{% endset %}
139+
140+
{% set magiceden_trades %}
141+
ALTER VIEW magiceden.trades SET TBLPROPERTIES('dune.public'='true',
142+
'dune.data_explorer.blockchains'='["solana"]',
143+
'dune.data_explorer.category'='abstraction',
144+
'dune.data_explorer.abstraction.type'='project',
145+
'dune.data_explorer.abstraction.name'='magiceden',
146+
'dune.data_explorer.contributors'='["soispoke"]');
147+
{% endset %}
148+
149+
{% set magiceden_mints %}
150+
ALTER VIEW magiceden.mints SET TBLPROPERTIES('dune.public'='true',
151+
'dune.data_explorer.blockchains'='["solana"]',
152+
'dune.data_explorer.category'='abstraction',
153+
'dune.data_explorer.abstraction.type'='project',
154+
'dune.data_explorer.abstraction.name'='magiceden',
155+
'dune.data_explorer.contributors'='["soispoke"]');
156+
{% endset %}
157+
158+
{% set nft_transactions %}
159+
ALTER TABLE nft.transactions SET TBLPROPERTIES('dune.public'='true',
160+
'dune.data_explorer.blockchains'='["ethereum","solana"]',
161+
'dune.data_explorer.category'='abstraction',
162+
'dune.data_explorer.abstraction.type'='sector',
163+
'dune.data_explorer.abstraction.name'='nft',
164+
'dune.data_explorer.contributors'='["soispoke"]');
165+
{% endset %}
166+
167+
{% set nft_trades %}
168+
ALTER TABLE nft.trades SET TBLPROPERTIES('dune.public'='true',
169+
'dune.data_explorer.blockchains'='["ethereum","solana"]',
170+
'dune.data_explorer.category'='abstraction',
171+
'dune.data_explorer.abstraction.type'='sector',
172+
'dune.data_explorer.abstraction.name'='nft',
173+
'dune.data_explorer.contributors'='["soispoke"]');
174+
{% endset %}
175+
176+
{% set nft_mints %}
177+
ALTER TABLE nft.mints SET TBLPROPERTIES('dune.public'='true',
178+
'dune.data_explorer.blockchains'='["ethereum","solana"]',
179+
'dune.data_explorer.category'='abstraction',
180+
'dune.data_explorer.abstraction.type'='sector',
181+
'dune.data_explorer.abstraction.name'='nft',
182+
'dune.data_explorer.contributors'='["soispoke"]');
183+
{% endset %}
184+
185+
{% set nft_burns %}
186+
ALTER TABLE nft.burns SET TBLPROPERTIES('dune.public'='true',
187+
'dune.data_explorer.blockchains'='["ethereum","solana"]',
188+
'dune.data_explorer.category'='abstraction',
189+
'dune.data_explorer.abstraction.type'='sector',
190+
'dune.data_explorer.abstraction.name'='nft',
191+
'dune.data_explorer.contributors'='["soispoke"]');
192+
{% endset %}
193+
194+
{% set nft_fees %}
195+
ALTER TABLE nft.fees SET TBLPROPERTIES('dune.public'='true',
196+
'dune.data_explorer.blockchains'='["ethereum","solana"]',
197+
'dune.data_explorer.category'='abstraction',
198+
'dune.data_explorer.abstraction.type'='sector',
199+
'dune.data_explorer.abstraction.name'='nft',
200+
'dune.data_explorer.contributors'='["soispoke"]');
137201
{% endset %}
138202

139203
{% set tokens_ethereum_erc20 %}
@@ -226,12 +290,19 @@ ALTER VIEW ens.view_renewals SET TBLPROPERTIES('dune.public'='true',
226290
{% do run_query(balances_ethereum_erc1155_day) %}
227291
{% do run_query(balances_ethereum_erc1155_hour) %}
228292
{% do run_query(balances_ethereum_erc1155_latest) %}
293+
{% do run_query(opensea_transactions) %}
294+
{% do run_query(opensea_trades) %}
295+
{% do run_query(opensea_mints) %}
296+
{% do run_query(opensea_burns) %}
297+
{% do run_query(opensea_fees) %}
298+
{% do run_query(magiceden_transactions) %}
229299
{% do run_query(magiceden_trades) %}
300+
{% do run_query(magiceden_mints) %}
301+
{% do run_query(nft_transactions) %}
230302
{% do run_query(nft_trades) %}
231-
{% do run_query(opensea_active_traders_day) %}
232-
{% do run_query(opensea_trades) %}
233-
{% do run_query(opensea_txns_day) %}
234-
{% do run_query(opensea_volume_day) %}
303+
{% do run_query(nft_mints) %}
304+
{% do run_query(nft_burns) %}
305+
{% do run_query(nft_fees) %}
235306
{% do run_query(tokens_ethereum_erc20) %}
236307
{% do run_query(transfers_ethereum_erc20) %}
237308
{% do run_query(tokens_ethereum_nft) %}

‎spellbook/macros/optimize_tables.sql‎

Lines changed: 29 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -24,33 +24,53 @@ OPTIMIZE transfers_ethereum.erc1155_agg_hour;
2424
OPTIMIZE transfers_ethereum.erc1155_agg_day;
2525
{% endset %}
2626

27-
{% set opensea_trades %}
28-
OPTIMIZE opensea.trades;
27+
{% set opensea_transactions %}
28+
OPTIMIZE opensea.transactions;
2929
{% endset %}
3030

31-
{% set magiceden_trades %}
32-
OPTIMIZE magiceden.trades;
31+
{% set magiceden_transactions %}
32+
OPTIMIZE magiceden.transactions;
3333
{% endset %}
3434

35-
{% set uniswap_trades %}
36-
OPTIMIZE uniswap.trades;
35+
{% set nft_transactions %}
36+
OPTIMIZE nft.transactions;
3737
{% endset %}
3838

3939
{% set nft_trades %}
4040
OPTIMIZE nft.trades;
4141
{% endset %}
4242

43+
{% set nft_mints %}
44+
OPTIMIZE nft.mints;
45+
{% endset %}
46+
47+
{% set nft_burns %}
48+
OPTIMIZE nft.burns;
49+
{% endset %}
50+
51+
{% set nft_fees %}
52+
OPTIMIZE nft.fees;
53+
{% endset %}
54+
55+
{% set uniswap_trades %}
56+
OPTIMIZE uniswap.trades;
57+
{% endset %}
4358

4459
{% do run_query(transfers_ethereum_erc20_agg_hour) %}
4560
{% do run_query(transfers_ethereum_erc20_agg_day) %}
4661
{% do run_query(transfers_ethereum_erc721_agg_hour) %}
4762
{% do run_query(transfers_ethereum_erc721_agg_day) %}
4863
{% do run_query(transfers_ethereum_erc1155_agg_hour) %}
4964
{% do run_query(transfers_ethereum_erc1155_agg_day) %}
50-
{% do run_query(opensea_trades) %}
51-
{% do run_query(magiceden_trades) %}
52-
{% do run_query(uniswap_trades) %}
65+
{% do run_query(opensea_transactions) %}
66+
{% do run_query(magiceden_transactions) %}
67+
{% do run_query(nft_transactions) %}
5368
{% do run_query(nft_trades) %}
69+
{% do run_query(nft_mints) %}
70+
{% do run_query(nft_burns) %}
71+
{% do run_query(nft_fees) %}
72+
{% do run_query(uniswap_trades) %}
73+
5474

5575
{% do log("Tables Optimized", info=True) %}
5676
{%- else -%}

‎spellbook/models/magiceden/magiceden_mints.sql‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
{{ config(
2+
alias ='mints'
3+
)
4+
}}
5+
6+
SELECT blockchain,
7+
project,
8+
version,
9+
block_time,
10+
NULL::string as token_id,
11+
NULL::string as collection,
12+
amount_usd,
13+
token_standard,
14+
NULL::string as trade_type,
15+
NULL::string as number_of_items,
16+
NULL::string as trade_category,
17+
evt_type,
18+
NULL::string as seller,
19+
NULL::string as buyer,
20+
amount_original,
21+
amount_raw,
22+
currency_symbol,
23+
currency_contract,
24+
NULL::string as nft_contract_address,
25+
project_contract_address,
26+
NULL::string as aggregator_name,
27+
NULL::string as aggregator_address,
28+
tx_hash,
29+
NULL::string as tx_from,
30+
NULL::string as tx_to,
31+
unique_trade_id
32+
FROM {{ ref('magiceden_solana_mints') }}

0 commit comments

Comments
 (0)