diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..91006c3f6 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,49 @@ +# Contributing to The Mempool Open Source Project + +Thank you for contributing to The Mempool Open Source Project managed by Mempool Space K.K. (“Mempool”). + +In order to clarify the intellectual property license granted with Contributions from any person or entity, Mempool must have a statement on file from each Contributor indicating their agreement to the Contributor License Agreement (“Agreement”). This license is for your protection as a Contributor as well as the protection of Mempool and its other contributors and users; it does not change your rights to use your own Contributions for any other purpose. + +When submitting a pull request for the first time, please create a file with a name like `/contributors/{github_username}.txt`, and in the content of that file indicate your agreement to the Contributor License Agreement terms below. An example of what that file should contain can be seen in wiz's agreement file. (This method of CLA "signing" is borrowed from Medium's open source project.) + +# Contributor License Agreement + +Last Updated: January 25, 2022 + +By accepting this Agreement, You agree to the following terms and conditions for Your present and future Contributions submitted to Mempool. Except for the license granted herein to Mempool and recipients of software distributed by Mempool, You reserve all right, title, and interest in and to Your Contributions. + +### 1. Definitions + +“You” (or “Your”) shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with Mempool. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, “control” means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + +“Contribution” shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to Mempool for inclusion in, or documentation of, any of the products owned or managed by Mempool (“Work”). For the purposes of this definition, “submitted” means any form of electronic, verbal, or written communication sent to Mempool or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, Mempool for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as “Not a Contribution.” + +### 2. Grant of Copyright License + +Subject to the terms and conditions of this Agreement, You hereby grant to Mempool and to recipients of software distributed by Mempool a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. + +### 3. Grant of Patent License + +Subject to the terms and conditions of this Agreement, You hereby grant to Mempool and to recipients of software distributed by Mempool a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. + +### 4. Authority + +You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to Mempool, or that your employer has executed a separate Corporate Contributor License Agreement with Mempool. + +### 5. Originality + +You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware, and which are associated with any part of Your Contributions. + +### 6. Support + +You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + +### 7. Third Party Contributions + +Should You wish to submit work that is not Your original creation, You may submit it to Mempool separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as “Submitted on behalf of a third-party: [named here]”. + +### 8. Notifications + +You agree to notify Mempool of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect. + +EOF diff --git a/LICENSE b/LICENSE index 1826f463d..966417847 100644 --- a/LICENSE +++ b/LICENSE @@ -1,5 +1,5 @@ The Mempool Open Source Project -Copyright (c) 2019-2021 The Mempool Open Source Project Developers +Copyright (c) 2019-2022 The Mempool Open Source Project Developers This program is free software; you can redistribute it and/or modify it under the terms of (at your option) either: diff --git a/backend/README.md b/backend/README.md new file mode 100644 index 000000000..e65205d67 --- /dev/null +++ b/backend/README.md @@ -0,0 +1,22 @@ +# Setup backend watchers + +The backend is static. Typescript scripts are compiled into the `dist` folder and served through a node web server. + +You can avoid the manual shutdown/recompile/restart command line cycle by using a watcher. + +Make sure you are in the `backend` directory `cd backend`. + +1. Install nodemon and ts-node + +``` +sudo npm install -g ts-node nodemon +``` + +2. Run the watcher + +> Note: You can find your npm global binary folder using `npm -g bin`, where nodemon will be installed. + +``` +nodemon src/index.ts --ignore cache/ --ignore pools.json +``` + diff --git a/backend/mempool-config.sample.json b/backend/mempool-config.sample.json index ea656c1de..8a9295b3a 100644 --- a/backend/mempool-config.sample.json +++ b/backend/mempool-config.sample.json @@ -12,9 +12,12 @@ "BLOCK_WEIGHT_UNITS": 4000000, "INITIAL_BLOCKS_AMOUNT": 8, "MEMPOOL_BLOCKS_AMOUNT": 8, + "INDEXING_BLOCKS_AMOUNT": 1100, "PRICE_FEED_UPDATE_INTERVAL": 3600, "USE_SECOND_NODE_FOR_MINFEE": false, - "EXTERNAL_ASSETS": [] + "EXTERNAL_ASSETS": [ + "https://mempool.space/resources/pools.json" + ] }, "CORE_RPC": { "HOST": "127.0.0.1", diff --git a/backend/src/api/bitcoin/bitcoin-api.ts b/backend/src/api/bitcoin/bitcoin-api.ts index b0a04116f..9950cdcb3 100644 --- a/backend/src/api/bitcoin/bitcoin-api.ts +++ b/backend/src/api/bitcoin/bitcoin-api.ts @@ -107,14 +107,25 @@ class BitcoinApi implements AbstractBitcoinApi { const outSpends: IEsploraApi.Outspend[] = []; const tx = await this.$getRawTransaction(txId, true, false); for (let i = 0; i < tx.vout.length; i++) { - const txOut = await this.bitcoindClient.getTxOut(txId, i); - outSpends.push({ - spent: txOut === null, - }); + if (tx.status && tx.status.block_height == 0) { + outSpends.push({ + spent: false + }); + } else { + const txOut = await this.bitcoindClient.getTxOut(txId, i); + outSpends.push({ + spent: txOut === null, + }); + } } return outSpends; } + $getEstimatedHashrate(blockHeight: number): Promise { + // 120 is the default block span in Core + return this.bitcoindClient.getNetworkHashPs(120, blockHeight); + } + protected async $convertTransaction(transaction: IBitcoinApi.Transaction, addPrevout: boolean): Promise { let esploraTransaction: IEsploraApi.Transaction = { txid: transaction.txid, diff --git a/backend/src/api/blocks.ts b/backend/src/api/blocks.ts index 1043c344f..22bea2480 100644 --- a/backend/src/api/blocks.ts +++ b/backend/src/api/blocks.ts @@ -2,11 +2,14 @@ import config from '../config'; import bitcoinApi from './bitcoin/bitcoin-api-factory'; import logger from '../logger'; import memPool from './mempool'; -import { BlockExtended, TransactionExtended } from '../mempool.interfaces'; +import { BlockExtended, PoolTag, TransactionExtended, TransactionMinerInfo } from '../mempool.interfaces'; import { Common } from './common'; import diskCache from './disk-cache'; import transactionUtils from './transaction-utils'; import bitcoinClient from './bitcoin/bitcoin-client'; +import { IEsploraApi } from './bitcoin/esplora-api.interface'; +import poolsRepository from '../repositories/PoolsRepository'; +import blocksRepository from '../repositories/BlocksRepository'; class Blocks { private blocks: BlockExtended[] = []; @@ -15,6 +18,7 @@ class Blocks { private lastDifficultyAdjustmentTime = 0; private previousDifficultyRetarget = 0; private newBlockCallbacks: ((block: BlockExtended, txIds: string[], transactions: TransactionExtended[]) => void)[] = []; + private blockIndexingStarted = false; constructor() { } @@ -30,6 +34,186 @@ class Blocks { this.newBlockCallbacks.push(fn); } + /** + * Return the list of transaction for a block + * @param blockHash + * @param blockHeight + * @param onlyCoinbase - Set to true if you only need the coinbase transaction + * @returns Promise + */ + private async $getTransactionsExtended(blockHash: string, blockHeight: number, onlyCoinbase: boolean): Promise { + const transactions: TransactionExtended[] = []; + const txIds: string[] = await bitcoinApi.$getTxIdsForBlock(blockHash); + + const mempool = memPool.getMempool(); + let transactionsFound = 0; + let transactionsFetched = 0; + + for (let i = 0; i < txIds.length; i++) { + if (mempool[txIds[i]]) { + // We update blocks before the mempool (index.ts), therefore we can + // optimize here by directly fetching txs in the "outdated" mempool + transactions.push(mempool[txIds[i]]); + transactionsFound++; + } else if (config.MEMPOOL.BACKEND === 'esplora' || memPool.isInSync() || i === 0) { + // Otherwise we fetch the tx data through backend services (esplora, electrum, core rpc...) + if (i % (Math.round((txIds.length) / 10)) === 0 || i + 1 === txIds.length) { // Avoid log spam + logger.debug(`Indexing tx ${i + 1} of ${txIds.length} in block #${blockHeight}`); + } + try { + const tx = await transactionUtils.$getTransactionExtended(txIds[i]); + transactions.push(tx); + transactionsFetched++; + } catch (e) { + logger.debug('Error fetching block tx: ' + (e instanceof Error ? e.message : e)); + if (i === 0) { + throw new Error('Failed to fetch Coinbase transaction: ' + txIds[i]); + } + } + } + + if (onlyCoinbase === true) { + break; // Fetch the first transaction and exit + } + } + + transactions.forEach((tx) => { + if (!tx.cpfpChecked) { + Common.setRelativesAndGetCpfpInfo(tx, mempool); // Child Pay For Parent + } + }); + + logger.debug(`${transactionsFound} of ${txIds.length} found in mempool. ${transactionsFetched} fetched through backend service.`); + + return transactions; + } + + /** + * Return a block with additional data (reward, coinbase, fees...) + * @param block + * @param transactions + * @returns BlockExtended + */ + private getBlockExtended(block: IEsploraApi.Block, transactions: TransactionExtended[]): BlockExtended { + const blockExtended: BlockExtended = Object.assign({}, block); + blockExtended.reward = transactions[0].vout.reduce((acc, curr) => acc + curr.value, 0); + blockExtended.coinbaseTx = transactionUtils.stripCoinbaseTransaction(transactions[0]); + + const transactionsTmp = [...transactions]; + transactionsTmp.shift(); + transactionsTmp.sort((a, b) => b.effectiveFeePerVsize - a.effectiveFeePerVsize); + blockExtended.medianFee = transactionsTmp.length > 0 ? Common.median(transactionsTmp.map((tx) => tx.effectiveFeePerVsize)) : 0; + blockExtended.feeRange = transactionsTmp.length > 0 ? Common.getFeesInRange(transactionsTmp, 8) : [0, 0]; + + return blockExtended; + } + + /** + * Try to find which miner found the block + * @param txMinerInfo + * @returns + */ + private async $findBlockMiner(txMinerInfo: TransactionMinerInfo | undefined): Promise { + if (txMinerInfo === undefined || txMinerInfo.vout.length < 1) { + return await poolsRepository.$getUnknownPool(); + } + + const asciiScriptSig = transactionUtils.hex2ascii(txMinerInfo.vin[0].scriptsig); + const address = txMinerInfo.vout[0].scriptpubkey_address; + + const pools: PoolTag[] = await poolsRepository.$getPools(); + for (let i = 0; i < pools.length; ++i) { + if (address !== undefined) { + const addresses: string[] = JSON.parse(pools[i].addresses); + if (addresses.indexOf(address) !== -1) { + return pools[i]; + } + } + + const regexes: string[] = JSON.parse(pools[i].regexes); + for (let y = 0; y < regexes.length; ++y) { + const match = asciiScriptSig.match(regexes[y]); + if (match !== null) { + return pools[i]; + } + } + } + + return await poolsRepository.$getUnknownPool(); + } + + /** + * Index all blocks metadata for the mining dashboard + */ + public async $generateBlockDatabase() { + if (['mainnet', 'testnet', 'signet'].includes(config.MEMPOOL.NETWORK) === false || // Bitcoin only + config.MEMPOOL.INDEXING_BLOCKS_AMOUNT === 0 || // Indexing must be enabled + !memPool.isInSync() || // We sync the mempool first + this.blockIndexingStarted === true // Indexing must not already be in progress + ) { + return; + } + + const blockchainInfo = await bitcoinClient.getBlockchainInfo(); + if (blockchainInfo.blocks !== blockchainInfo.headers) { + return; + } + + this.blockIndexingStarted = true; + + try { + let currentBlockHeight = blockchainInfo.blocks; + + let indexingBlockAmount = config.MEMPOOL.INDEXING_BLOCKS_AMOUNT; + if (indexingBlockAmount <= -1) { + indexingBlockAmount = currentBlockHeight + 1; + } + + const lastBlockToIndex = Math.max(0, currentBlockHeight - indexingBlockAmount + 1); + + logger.info(`Indexing blocks from #${currentBlockHeight} to #${lastBlockToIndex}`); + + const chunkSize = 10000; + while (currentBlockHeight >= lastBlockToIndex) { + const endBlock = Math.max(0, lastBlockToIndex, currentBlockHeight - chunkSize + 1); + + const missingBlockHeights: number[] = await blocksRepository.$getMissingBlocksBetweenHeights( + currentBlockHeight, endBlock); + if (missingBlockHeights.length <= 0) { + logger.debug(`No missing blocks between #${currentBlockHeight} to #${endBlock}`); + currentBlockHeight -= chunkSize; + continue; + } + + logger.debug(`Indexing ${missingBlockHeights.length} blocks from #${currentBlockHeight} to #${endBlock}`); + + for (const blockHeight of missingBlockHeights) { + if (blockHeight < lastBlockToIndex) { + break; + } + try { + logger.debug(`Indexing block #${blockHeight}`); + const blockHash = await bitcoinApi.$getBlockHash(blockHeight); + const block = await bitcoinApi.$getBlock(blockHash); + const transactions = await this.$getTransactionsExtended(blockHash, block.height, true); + const blockExtended = this.getBlockExtended(block, transactions); + const miner = await this.$findBlockMiner(blockExtended.coinbaseTx); + const coinbase: IEsploraApi.Transaction = await bitcoinApi.$getRawTransaction(transactions[0].txid, true); + await blocksRepository.$saveBlockInDatabase(blockExtended, blockHash, coinbase.hex, miner); + } catch (e) { + logger.err(`Something went wrong while indexing blocks.` + e); + } + } + + currentBlockHeight -= chunkSize; + } + logger.info('Block indexing completed'); + } catch (e) { + logger.err('An error occured in $generateBlockDatabase(). Skipping block indexing. ' + e); + console.log(e); + } + } + public async $updateBlocks() { const blockHeightTip = await bitcoinApi.$getBlockHeightTip(); @@ -70,49 +254,18 @@ class Blocks { logger.debug(`New block found (#${this.currentBlockHeight})!`); } - const transactions: TransactionExtended[] = []; - const blockHash = await bitcoinApi.$getBlockHash(this.currentBlockHeight); const block = await bitcoinApi.$getBlock(blockHash); const txIds: string[] = await bitcoinApi.$getTxIdsForBlock(blockHash); + const transactions = await this.$getTransactionsExtended(blockHash, block.height, false); + const blockExtended: BlockExtended = this.getBlockExtended(block, transactions); + const coinbase: IEsploraApi.Transaction = await bitcoinApi.$getRawTransaction(transactions[0].txid, true); - const mempool = memPool.getMempool(); - let transactionsFound = 0; - - for (let i = 0; i < txIds.length; i++) { - if (mempool[txIds[i]]) { - transactions.push(mempool[txIds[i]]); - transactionsFound++; - } else if (config.MEMPOOL.BACKEND === 'esplora' || memPool.isInSync() || i === 0) { - logger.debug(`Fetching block tx ${i} of ${txIds.length}`); - try { - const tx = await transactionUtils.$getTransactionExtended(txIds[i]); - transactions.push(tx); - } catch (e) { - logger.debug('Error fetching block tx: ' + (e instanceof Error ? e.message : e)); - if (i === 0) { - throw new Error('Failed to fetch Coinbase transaction: ' + txIds[i]); - } - } - } + if (['mainnet', 'testnet', 'signet'].includes(config.MEMPOOL.NETWORK) === true) { + const miner = await this.$findBlockMiner(blockExtended.coinbaseTx); + await blocksRepository.$saveBlockInDatabase(blockExtended, blockHash, coinbase.hex, miner); } - transactions.forEach((tx) => { - if (!tx.cpfpChecked) { - Common.setRelativesAndGetCpfpInfo(tx, mempool); - } - }); - - logger.debug(`${transactionsFound} of ${txIds.length} found in mempool. ${txIds.length - transactionsFound} not found.`); - - const blockExtended: BlockExtended = Object.assign({}, block); - blockExtended.reward = transactions[0].vout.reduce((acc, curr) => acc + curr.value, 0); - blockExtended.coinbaseTx = transactionUtils.stripCoinbaseTransaction(transactions[0]); - transactions.shift(); - transactions.sort((a, b) => b.effectiveFeePerVsize - a.effectiveFeePerVsize); - blockExtended.medianFee = transactions.length > 0 ? Common.median(transactions.map((tx) => tx.effectiveFeePerVsize)) : 0; - blockExtended.feeRange = transactions.length > 0 ? Common.getFeesInRange(transactions, 8) : [0, 0]; - if (block.height % 2016 === 0) { this.previousDifficultyRetarget = (block.difficulty - this.currentDifficulty) / this.currentDifficulty * 100; this.lastDifficultyAdjustmentTime = block.timestamp; @@ -130,6 +283,8 @@ class Blocks { if (memPool.isInSync()) { diskCache.$saveCacheToDisk(); } + + return; } } diff --git a/backend/src/api/database-migration.ts b/backend/src/api/database-migration.ts index 2ac97636e..24ecc03cf 100644 --- a/backend/src/api/database-migration.ts +++ b/backend/src/api/database-migration.ts @@ -3,10 +3,10 @@ import config from '../config'; import { DB } from '../database'; import logger from '../logger'; -const sleep = (ms: number) => new Promise( res => setTimeout(res, ms)); +const sleep = (ms: number) => new Promise(res => setTimeout(res, ms)); class DatabaseMigration { - private static currentVersion = 2; + private static currentVersion = 4; private queryTimeout = 120000; private statisticsAddedIndexed = false; @@ -83,6 +83,13 @@ class DatabaseMigration { if (databaseSchemaVersion < 2 && this.statisticsAddedIndexed === false) { await this.$executeQuery(connection, `CREATE INDEX added ON statistics (added);`); } + if (databaseSchemaVersion < 3) { + await this.$executeQuery(connection, this.getCreatePoolsTableQuery(), await this.$checkIfTableExists('pools')); + } + if (databaseSchemaVersion < 4) { + await this.$executeQuery(connection, 'DROP table IF EXISTS blocks;'); + await this.$executeQuery(connection, this.getCreateBlocksTableQuery(), await this.$checkIfTableExists('blocks')); + } connection.release(); } catch (e) { connection.release(); @@ -197,7 +204,6 @@ class DatabaseMigration { const connection = await DB.pool.getConnection(); try { await this.$executeQuery(connection, 'START TRANSACTION;'); - await this.$executeQuery(connection, 'SET autocommit = 0;'); for (const query of transactionQueries) { await this.$executeQuery(connection, query); } @@ -335,6 +341,37 @@ class DatabaseMigration { final_tx int(11) NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8;`; } + + private getCreatePoolsTableQuery(): string { + return `CREATE TABLE IF NOT EXISTS pools ( + id int(11) NOT NULL AUTO_INCREMENT, + name varchar(50) NOT NULL, + link varchar(255) NOT NULL, + addresses text NOT NULL, + regexes text NOT NULL, + PRIMARY KEY (id) + ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;`; + } + + private getCreateBlocksTableQuery(): string { + return `CREATE TABLE IF NOT EXISTS blocks ( + height int(11) unsigned NOT NULL, + hash varchar(65) NOT NULL, + blockTimestamp timestamp NOT NULL, + size int(11) unsigned NOT NULL, + weight int(11) unsigned NOT NULL, + tx_count int(11) unsigned NOT NULL, + coinbase_raw text, + difficulty bigint(20) unsigned NOT NULL, + pool_id int(11) DEFAULT -1, + fees double unsigned NOT NULL, + fee_span json NOT NULL, + median_fee double unsigned NOT NULL, + PRIMARY KEY (height), + INDEX (pool_id), + FOREIGN KEY (pool_id) REFERENCES pools (id) + ) ENGINE=InnoDB DEFAULT CHARSET=utf8;`; + } } export default new DatabaseMigration(); diff --git a/backend/src/api/mining.ts b/backend/src/api/mining.ts new file mode 100644 index 000000000..c89ea9324 --- /dev/null +++ b/backend/src/api/mining.ts @@ -0,0 +1,69 @@ +import { PoolInfo, PoolStats } from '../mempool.interfaces'; +import BlocksRepository, { EmptyBlocks } from '../repositories/BlocksRepository'; +import PoolsRepository from '../repositories/PoolsRepository'; +import bitcoinClient from './bitcoin/bitcoin-client'; + +class Mining { + constructor() { + } + + /** + * Generate high level overview of the pool ranks and general stats + */ + public async $getPoolsStats(interval: string | null) : Promise { + let sqlInterval: string | null = null; + switch (interval) { + case '24h': sqlInterval = '1 DAY'; break; + case '3d': sqlInterval = '3 DAY'; break; + case '1w': sqlInterval = '1 WEEK'; break; + case '1m': sqlInterval = '1 MONTH'; break; + case '3m': sqlInterval = '3 MONTH'; break; + case '6m': sqlInterval = '6 MONTH'; break; + case '1y': sqlInterval = '1 YEAR'; break; + case '2y': sqlInterval = '2 YEAR'; break; + case '3y': sqlInterval = '3 YEAR'; break; + default: sqlInterval = null; break; + } + + const poolsStatistics = {}; + + const poolsInfo: PoolInfo[] = await PoolsRepository.$getPoolsInfo(sqlInterval); + const emptyBlocks: EmptyBlocks[] = await BlocksRepository.$countEmptyBlocks(sqlInterval); + + const poolsStats: PoolStats[] = []; + let rank = 1; + + poolsInfo.forEach((poolInfo: PoolInfo) => { + const poolStat: PoolStats = { + poolId: poolInfo.poolId, // mysql row id + name: poolInfo.name, + link: poolInfo.link, + blockCount: poolInfo.blockCount, + rank: rank++, + emptyBlocks: 0, + } + for (let i = 0; i < emptyBlocks.length; ++i) { + if (emptyBlocks[i].poolId === poolInfo.poolId) { + poolStat.emptyBlocks++; + } + } + poolsStats.push(poolStat); + }); + + poolsStatistics['pools'] = poolsStats; + + const oldestBlock = new Date(await BlocksRepository.$oldestBlockTimestamp()); + poolsStatistics['oldestIndexedBlockTimestamp'] = oldestBlock.getTime(); + + const blockCount: number = await BlocksRepository.$blockCount(sqlInterval); + poolsStatistics['blockCount'] = blockCount; + + const blockHeightTip = await bitcoinClient.getBlockCount(); + const lastBlockHashrate = await bitcoinClient.getNetworkHashPs(120, blockHeightTip); + poolsStatistics['lastEstimatedHashrate'] = lastBlockHashrate; + + return poolsStatistics; + } +} + +export default new Mining(); diff --git a/backend/src/api/pools-parser.ts b/backend/src/api/pools-parser.ts new file mode 100644 index 000000000..194ce0dd9 --- /dev/null +++ b/backend/src/api/pools-parser.ts @@ -0,0 +1,173 @@ +import { readFileSync } from 'fs'; +import { DB } from '../database'; +import logger from '../logger'; +import config from '../config'; + +interface Pool { + name: string; + link: string; + regexes: string[]; + addresses: string[]; +} + +class PoolsParser { + /** + * Parse the pools.json file, consolidate the data and dump it into the database + */ + public async migratePoolsJson() { + if (['mainnet', 'testnet', 'signet'].includes(config.MEMPOOL.NETWORK) === false) { + return; + } + + logger.debug('Importing pools.json to the database, open ./pools.json'); + + let poolsJson: object = {}; + try { + const fileContent: string = readFileSync('./pools.json', 'utf8'); + poolsJson = JSON.parse(fileContent); + } catch (e) { + logger.err('Unable to open ./pools.json, does the file exist?'); + await this.insertUnknownPool(); + return; + } + + // First we save every entries without paying attention to pool duplication + const poolsDuplicated: Pool[] = []; + + logger.debug('Parse coinbase_tags'); + const coinbaseTags = Object.entries(poolsJson['coinbase_tags']); + for (let i = 0; i < coinbaseTags.length; ++i) { + poolsDuplicated.push({ + 'name': (coinbaseTags[i][1]).name, + 'link': (coinbaseTags[i][1]).link, + 'regexes': [coinbaseTags[i][0]], + 'addresses': [], + }); + } + logger.debug('Parse payout_addresses'); + const addressesTags = Object.entries(poolsJson['payout_addresses']); + for (let i = 0; i < addressesTags.length; ++i) { + poolsDuplicated.push({ + 'name': (addressesTags[i][1]).name, + 'link': (addressesTags[i][1]).link, + 'regexes': [], + 'addresses': [addressesTags[i][0]], + }); + } + + // Then, we find unique mining pool names + logger.debug('Identify unique mining pools'); + const poolNames: string[] = []; + for (let i = 0; i < poolsDuplicated.length; ++i) { + if (poolNames.indexOf(poolsDuplicated[i].name) === -1) { + poolNames.push(poolsDuplicated[i].name); + } + } + logger.debug(`Found ${poolNames.length} unique mining pools`); + + // Get existing pools from the db + const connection = await DB.pool.getConnection(); + let existingPools; + try { + [existingPools] = await connection.query({ sql: 'SELECT * FROM pools;', timeout: 120000 }); + } catch (e) { + logger.err('Unable to get existing pools from the database, skipping pools.json import'); + connection.release(); + return; + } + + // Finally, we generate the final consolidated pools data + const finalPoolDataAdd: Pool[] = []; + const finalPoolDataUpdate: Pool[] = []; + for (let i = 0; i < poolNames.length; ++i) { + let allAddresses: string[] = []; + let allRegexes: string[] = []; + const match = poolsDuplicated.filter((pool: Pool) => pool.name === poolNames[i]); + + for (let y = 0; y < match.length; ++y) { + allAddresses = allAddresses.concat(match[y].addresses); + allRegexes = allRegexes.concat(match[y].regexes); + } + + const finalPoolName = poolNames[i].replace(`'`, `''`); // To support single quote in names when doing db queries + + if (existingPools.find((pool) => pool.name === poolNames[i]) !== undefined) { + logger.debug(`Update '${finalPoolName}' mining pool`); + finalPoolDataUpdate.push({ + 'name': finalPoolName, + 'link': match[0].link, + 'regexes': allRegexes, + 'addresses': allAddresses, + }); + } else { + logger.debug(`Add '${finalPoolName}' mining pool`); + finalPoolDataAdd.push({ + 'name': finalPoolName, + 'link': match[0].link, + 'regexes': allRegexes, + 'addresses': allAddresses, + }); + } + } + + logger.debug(`Update pools table now`); + + // Add new mining pools into the database + let queryAdd: string = 'INSERT INTO pools(name, link, regexes, addresses) VALUES '; + for (let i = 0; i < finalPoolDataAdd.length; ++i) { + queryAdd += `('${finalPoolDataAdd[i].name}', '${finalPoolDataAdd[i].link}', + '${JSON.stringify(finalPoolDataAdd[i].regexes)}', '${JSON.stringify(finalPoolDataAdd[i].addresses)}'),`; + } + queryAdd = queryAdd.slice(0, -1) + ';'; + + // Add new mining pools into the database + const updateQueries: string[] = []; + for (let i = 0; i < finalPoolDataUpdate.length; ++i) { + updateQueries.push(` + UPDATE pools + SET name='${finalPoolDataUpdate[i].name}', link='${finalPoolDataUpdate[i].link}', + regexes='${JSON.stringify(finalPoolDataUpdate[i].regexes)}', addresses='${JSON.stringify(finalPoolDataUpdate[i].addresses)}' + WHERE name='${finalPoolDataUpdate[i].name}' + ;`); + } + + try { + if (finalPoolDataAdd.length > 0) { + await connection.query({ sql: queryAdd, timeout: 120000 }); + } + for (const query of updateQueries) { + await connection.query({ sql: query, timeout: 120000 }); + } + await this.insertUnknownPool(); + connection.release(); + logger.info('Mining pools.json import completed'); + } catch (e) { + connection.release(); + logger.err(`Unable to import pools in the database!`); + throw e; + } + } + + /** + * Manually add the 'unknown pool' + */ + private async insertUnknownPool() { + const connection = await DB.pool.getConnection(); + try { + const [rows]: any[] = await connection.query({ sql: 'SELECT name from pools where name="Unknown"', timeout: 120000 }); + if (rows.length === 0) { + logger.debug('Manually inserting "Unknown" mining pool into the databse'); + await connection.query({ + sql: `INSERT INTO pools(name, link, regexes, addresses) + VALUES("Unknown", "https://learnmeabitcoin.com/technical/coinbase-transaction", "[]", "[]"); + `}); + } + } catch (e) { + logger.err('Unable to insert "Unknown" mining pool'); + } + + connection.release(); + } +} + +export default new PoolsParser(); diff --git a/backend/src/api/transaction-utils.ts b/backend/src/api/transaction-utils.ts index 1496b810b..2e669d709 100644 --- a/backend/src/api/transaction-utils.ts +++ b/backend/src/api/transaction-utils.ts @@ -44,6 +44,14 @@ class TransactionUtils { } return transactionExtended; } + + public hex2ascii(hex: string) { + let str = ''; + for (let i = 0; i < hex.length; i += 2) { + str += String.fromCharCode(parseInt(hex.substr(i, 2), 16)); + } + return str; + } } export default new TransactionUtils(); diff --git a/backend/src/config.ts b/backend/src/config.ts index 4c2888834..085d538c4 100644 --- a/backend/src/config.ts +++ b/backend/src/config.ts @@ -14,6 +14,7 @@ interface IConfig { BLOCK_WEIGHT_UNITS: number; INITIAL_BLOCKS_AMOUNT: number; MEMPOOL_BLOCKS_AMOUNT: number; + INDEXING_BLOCKS_AMOUNT: number; PRICE_FEED_UPDATE_INTERVAL: number; USE_SECOND_NODE_FOR_MINFEE: boolean; EXTERNAL_ASSETS: string[]; @@ -77,9 +78,12 @@ const defaults: IConfig = { 'BLOCK_WEIGHT_UNITS': 4000000, 'INITIAL_BLOCKS_AMOUNT': 8, 'MEMPOOL_BLOCKS_AMOUNT': 8, + 'INDEXING_BLOCKS_AMOUNT': 1100, // 0 = disable indexing, -1 = index all blocks 'PRICE_FEED_UPDATE_INTERVAL': 3600, 'USE_SECOND_NODE_FOR_MINFEE': false, - 'EXTERNAL_ASSETS': [], + 'EXTERNAL_ASSETS': [ + 'https://mempool.space/resources/pools.json' + ] }, 'ESPLORA': { 'REST_API_URL': 'http://127.0.0.1:3000', diff --git a/backend/src/index.ts b/backend/src/index.ts index f6615d1c8..f78c5922b 100644 --- a/backend/src/index.ts +++ b/backend/src/index.ts @@ -22,6 +22,7 @@ import loadingIndicators from './api/loading-indicators'; import mempool from './api/mempool'; import elementsParser from './api/liquid/elements-parser'; import databaseMigration from './api/database-migration'; +import poolsParser from './api/pools-parser'; import syncAssets from './sync-assets'; import icons from './api/liquid/icons'; import { Common } from './api/common'; @@ -88,6 +89,7 @@ class Server { await checkDbConnection(); try { await databaseMigration.$initializeOrMigrateDatabase(); + await poolsParser.migratePoolsJson(); } catch (e) { throw new Error(e instanceof Error ? e.message : 'Error'); } @@ -136,6 +138,8 @@ class Server { } await blocks.$updateBlocks(); await memPool.$updateMempool(); + blocks.$generateBlockDatabase(); + setTimeout(this.runMainUpdateLoop.bind(this), config.MEMPOOL.POLL_RATE_MS); this.currentBackendRetryInterval = 5; } catch (e) { @@ -252,6 +256,16 @@ class Server { .get(config.MEMPOOL.API_URL_PREFIX + 'statistics/1y', routes.$getStatisticsByTime.bind(routes, '1y')) .get(config.MEMPOOL.API_URL_PREFIX + 'statistics/2y', routes.$getStatisticsByTime.bind(routes, '2y')) .get(config.MEMPOOL.API_URL_PREFIX + 'statistics/3y', routes.$getStatisticsByTime.bind(routes, '3y')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/24h', routes.$getPools.bind(routes, '24h')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/3d', routes.$getPools.bind(routes, '3d')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/1w', routes.$getPools.bind(routes, '1w')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/1m', routes.$getPools.bind(routes, '1m')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/3m', routes.$getPools.bind(routes, '3m')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/6m', routes.$getPools.bind(routes, '6m')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/1y', routes.$getPools.bind(routes, '1y')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/2y', routes.$getPools.bind(routes, '2y')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/3y', routes.$getPools.bind(routes, '3y')) + .get(config.MEMPOOL.API_URL_PREFIX + 'mining/pools/all', routes.$getPools.bind(routes, 'all')) ; } diff --git a/backend/src/mempool.interfaces.ts b/backend/src/mempool.interfaces.ts index 2604a233c..5fb83d792 100644 --- a/backend/src/mempool.interfaces.ts +++ b/backend/src/mempool.interfaces.ts @@ -1,5 +1,25 @@ import { IEsploraApi } from './api/bitcoin/esplora-api.interface'; +export interface PoolTag { + id: number | null, // mysql row id + name: string, + link: string, + regexes: string, // JSON array + addresses: string, // JSON array +} + +export interface PoolInfo { + poolId: number, // mysql row id + name: string, + link: string, + blockCount: number, +} + +export interface PoolStats extends PoolInfo { + rank: number, + emptyBlocks: number, +} + export interface MempoolBlock { blockSize: number; blockVSize: number; diff --git a/backend/src/repositories/BlocksRepository.ts b/backend/src/repositories/BlocksRepository.ts new file mode 100644 index 000000000..947403e88 --- /dev/null +++ b/backend/src/repositories/BlocksRepository.ts @@ -0,0 +1,128 @@ +import { BlockExtended, PoolTag } from '../mempool.interfaces'; +import { DB } from '../database'; +import logger from '../logger'; + +export interface EmptyBlocks { + emptyBlocks: number; + poolId: number; +} + +class BlocksRepository { + /** + * Save indexed block data in the database + */ + public async $saveBlockInDatabase( + block: BlockExtended, + blockHash: string, + coinbaseHex: string | undefined, + poolTag: PoolTag + ) { + const connection = await DB.pool.getConnection(); + + try { + const query = `INSERT INTO blocks( + height, hash, blockTimestamp, size, + weight, tx_count, coinbase_raw, difficulty, + pool_id, fees, fee_span, median_fee + ) VALUE ( + ?, ?, FROM_UNIXTIME(?), ?, + ?, ?, ?, ?, + ?, ?, ?, ? + )`; + + const params: any[] = [ + block.height, blockHash, block.timestamp, block.size, + block.weight, block.tx_count, coinbaseHex ? coinbaseHex : '', block.difficulty, + poolTag.id, 0, '[]', block.medianFee, + ]; + + await connection.query(query, params); + } catch (e) { + logger.err('$saveBlockInDatabase() error' + (e instanceof Error ? e.message : e)); + } + + connection.release(); + } + + /** + * Get all block height that have not been indexed between [startHeight, endHeight] + */ + public async $getMissingBlocksBetweenHeights(startHeight: number, endHeight: number): Promise { + if (startHeight < endHeight) { + return []; + } + + const connection = await DB.pool.getConnection(); + const [rows] : any[] = await connection.query(` + SELECT height + FROM blocks + WHERE height <= ${startHeight} AND height >= ${endHeight} + ORDER BY height DESC; + `); + connection.release(); + + const indexedBlockHeights: number[] = []; + rows.forEach((row: any) => { indexedBlockHeights.push(row.height); }); + const seekedBlocks: number[] = Array.from(Array(startHeight - endHeight + 1).keys(), n => n + endHeight).reverse(); + const missingBlocksHeights = seekedBlocks.filter(x => indexedBlockHeights.indexOf(x) === -1); + + return missingBlocksHeights; + } + + /** + * Count empty blocks for all pools + */ + public async $countEmptyBlocks(interval: string | null): Promise { + const query = ` + SELECT pool_id as poolId + FROM blocks + WHERE tx_count = 1` + + (interval != null ? ` AND blockTimestamp BETWEEN DATE_SUB(NOW(), INTERVAL ${interval}) AND NOW()` : ``) + ; + + const connection = await DB.pool.getConnection(); + const [rows] = await connection.query(query); + connection.release(); + + return rows; + } + + /** + * Get blocks count for a period + */ + public async $blockCount(interval: string | null): Promise { + const query = ` + SELECT count(height) as blockCount + FROM blocks` + + (interval != null ? ` WHERE blockTimestamp BETWEEN DATE_SUB(NOW(), INTERVAL ${interval}) AND NOW()` : ``) + ; + + const connection = await DB.pool.getConnection(); + const [rows] = await connection.query(query); + connection.release(); + + return rows[0].blockCount; + } + + /** + * Get the oldest indexed block + */ + public async $oldestBlockTimestamp(): Promise { + const connection = await DB.pool.getConnection(); + const [rows]: any[] = await connection.query(` + SELECT blockTimestamp + FROM blocks + ORDER BY height + LIMIT 1; + `); + connection.release(); + + if (rows.length <= 0) { + return -1; + } + + return rows[0].blockTimestamp; + } +} + +export default new BlocksRepository(); \ No newline at end of file diff --git a/backend/src/repositories/PoolsRepository.ts b/backend/src/repositories/PoolsRepository.ts new file mode 100644 index 000000000..d1fb0da9a --- /dev/null +++ b/backend/src/repositories/PoolsRepository.ts @@ -0,0 +1,46 @@ +import { DB } from '../database'; +import { PoolInfo, PoolTag } from '../mempool.interfaces'; + +class PoolsRepository { + /** + * Get all pools tagging info + */ + public async $getPools(): Promise { + const connection = await DB.pool.getConnection(); + const [rows] = await connection.query('SELECT * FROM pools;'); + connection.release(); + return rows; + } + + /** + * Get unknown pool tagging info + */ + public async $getUnknownPool(): Promise { + const connection = await DB.pool.getConnection(); + const [rows] = await connection.query('SELECT * FROM pools where name = "Unknown"'); + connection.release(); + return rows[0]; + } + + /** + * Get basic pool info and block count + */ + public async $getPoolsInfo(interval: string | null): Promise { + const query = ` + SELECT COUNT(height) as blockCount, pool_id as poolId, pools.name as name, pools.link as link + FROM blocks + JOIN pools on pools.id = pool_id` + + (interval != null ? ` WHERE blocks.blockTimestamp BETWEEN DATE_SUB(NOW(), INTERVAL ${interval}) AND NOW()` : ``) + + ` GROUP BY pool_id + ORDER BY COUNT(height) DESC + `; + + const connection = await DB.pool.getConnection(); + const [rows] = await connection.query(query); + connection.release(); + + return rows; + } +} + +export default new PoolsRepository(); diff --git a/backend/src/routes.ts b/backend/src/routes.ts index 1d98c9f4e..044f9a3ac 100644 --- a/backend/src/routes.ts +++ b/backend/src/routes.ts @@ -20,6 +20,7 @@ import { Common } from './api/common'; import bitcoinClient from './api/bitcoin/bitcoin-client'; import elementsParser from './api/liquid/elements-parser'; import icons from './api/liquid/icons'; +import miningStats from './api/mining'; class Routes { constructor() {} @@ -531,6 +532,18 @@ class Routes { } } + public async $getPools(interval: string, req: Request, res: Response) { + try { + let stats = await miningStats.$getPoolsStats(interval); + res.header('Pragma', 'public'); + res.header('Cache-control', 'public'); + res.setHeader('Expires', new Date(Date.now() + 1000 * 60).toUTCString()); + res.json(stats); + } catch (e) { + res.status(500).send(e instanceof Error ? e.message : e); + } + } + public async getBlock(req: Request, res: Response) { try { const result = await bitcoinApi.$getBlock(req.params.hash); diff --git a/contributors/antonilol.txt b/contributors/antonilol.txt new file mode 100644 index 000000000..94f5ba0c6 --- /dev/null +++ b/contributors/antonilol.txt @@ -0,0 +1,3 @@ +I hereby accept the terms of the Contributor License Agreement in the CONTRIBUTING.md file of the mempool/mempool git repository as of January 25, 2022. + +Signed: antonilol diff --git a/contributors/emzy.txt b/contributors/emzy.txt new file mode 100644 index 000000000..7d0a9409e --- /dev/null +++ b/contributors/emzy.txt @@ -0,0 +1 @@ +Mempool Space K.K. has a signed CLA or other agreement on file with @emzy as of January 25, 2022 diff --git a/contributors/hunicus.txt b/contributors/hunicus.txt new file mode 100644 index 000000000..bf8f3ed15 --- /dev/null +++ b/contributors/hunicus.txt @@ -0,0 +1 @@ +Mempool Space K.K. has a signed CLA or other agreement on file with @hunicus as of January 25, 2022 diff --git a/contributors/knorrium.txt b/contributors/knorrium.txt new file mode 100644 index 000000000..144a966a4 --- /dev/null +++ b/contributors/knorrium.txt @@ -0,0 +1 @@ +Mempool Space K.K. has a signed CLA or other agreement on file with @knorrium as of January 25, 2022 diff --git a/contributors/miguelmedeiros.txt b/contributors/miguelmedeiros.txt new file mode 100644 index 000000000..3a7a2baab --- /dev/null +++ b/contributors/miguelmedeiros.txt @@ -0,0 +1 @@ +Mempool Space K.K. has a signed CLA or other agreement on file with @miguelmedeiros as of January 25, 2022 diff --git a/contributors/nymkappa.txt b/contributors/nymkappa.txt new file mode 100644 index 000000000..4e86a6f38 --- /dev/null +++ b/contributors/nymkappa.txt @@ -0,0 +1 @@ +Mempool Space K.K. has a signed CLA or other agreement on file with @nymkappa as of January 25, 2022 diff --git a/contributors/softsimon.txt b/contributors/softsimon.txt new file mode 100644 index 000000000..f33f99034 --- /dev/null +++ b/contributors/softsimon.txt @@ -0,0 +1,3 @@ +I hereby accept the terms of the Contributor License Agreement in the CONTRIBUTING.md file of the mempool/mempool git repository as of January 25, 2022. + +Signed: softsimon diff --git a/contributors/wiz.txt b/contributors/wiz.txt new file mode 100644 index 000000000..a006b4e04 --- /dev/null +++ b/contributors/wiz.txt @@ -0,0 +1,3 @@ +I hereby accept the terms of the Contributor License Agreement in the CONTRIBUTING.md file of the mempool/mempool git repository as of January 25, 2022. + +Signed: wiz diff --git a/docker/backend/start.sh b/docker/backend/start.sh index f982a18e5..372be346e 100644 --- a/docker/backend/start.sh +++ b/docker/backend/start.sh @@ -13,6 +13,7 @@ __MEMPOOL_RECOMMENDED_FEE_PERCENTILE__=${MEMPOOL_RECOMMENDED_FEE_PERCENTILE:=50} __MEMPOOL_BLOCK_WEIGHT_UNITS__=${MEMPOOL_BLOCK_WEIGHT_UNITS:=4000000} __MEMPOOL_INITIAL_BLOCKS_AMOUNT__=${MEMPOOL_INITIAL_BLOCKS_AMOUNT:=8} __MEMPOOL_MEMPOOL_BLOCKS_AMOUNT__=${MEMPOOL_MEMPOOL_BLOCKS_AMOUNT:=8} +__MEMPOOL_INDEXING_BLOCKS_AMOUNT__=${MEMPOOL_INDEXING_BLOCKS_AMOUNT:=1100} __MEMPOOL_PRICE_FEED_UPDATE_INTERVAL__=${MEMPOOL_PRICE_FEED_UPDATE_INTERVAL:=3600} __MEMPOOL_USE_SECOND_NODE_FOR_MINFEE__=${MEMPOOL_USE_SECOND_NODE_FOR_MINFEE:=false} __MEMPOOL_EXTERNAL_ASSETS__=${MEMPOOL_EXTERNAL_ASSETS:=[]} @@ -74,6 +75,7 @@ sed -i "s/__MEMPOOL_RECOMMENDED_FEE_PERCENTILE__/${__MEMPOOL_RECOMMENDED_FEE_PER sed -i "s/__MEMPOOL_BLOCK_WEIGHT_UNITS__/${__MEMPOOL_BLOCK_WEIGHT_UNITS__}/g" mempool-config.json sed -i "s/__MEMPOOL_INITIAL_BLOCKS_AMOUNT__/${__MEMPOOL_INITIAL_BLOCKS_AMOUNT__}/g" mempool-config.json sed -i "s/__MEMPOOL_MEMPOOL_BLOCKS_AMOUNT__/${__MEMPOOL_MEMPOOL_BLOCKS_AMOUNT__}/g" mempool-config.json +sed -i "s/__MEMPOOL_INDEXING_BLOCKS_AMOUNT__/${__MEMPOOL_INDEXING_BLOCKS_AMOUNT__}/g" mempool-config.json sed -i "s/__MEMPOOL_PRICE_FEED_UPDATE_INTERVAL__/${__MEMPOOL_PRICE_FEED_UPDATE_INTERVAL__}/g" mempool-config.json sed -i "s/__MEMPOOL_USE_SECOND_NODE_FOR_MINFEE__/${__MEMPOOL_USE_SECOND_NODE_FOR_MINFEE__}/g" mempool-config.json sed -i "s/__MEMPOOL_EXTERNAL_ASSETS__/${__MEMPOOL_EXTERNAL_ASSETS__}/g" mempool-config.json diff --git a/frontend/cypress/fixtures/pools.json b/frontend/cypress/fixtures/pools.json index ab198b6d0..1a69517e8 100644 --- a/frontend/cypress/fixtures/pools.json +++ b/frontend/cypress/fixtures/pools.json @@ -421,7 +421,7 @@ "link" : "http://www.dpool.top/" }, "/Rawpool.com/": { - "name" : "Rawpool.com", + "name" : "Rawpool", "link" : "https://www.rawpool.com/" }, "/haominer/": { @@ -488,10 +488,14 @@ "name" : "Binance Pool", "link" : "https://pool.binance.com/" }, - "/Minerium.com/" : { + "/Mined in the USA by: /Minerium.com/" : { "name" : "Minerium", "link" : "https://www.minerium.com/" }, + "/Minerium.com/" : { + "name" : "Minerium", + "link" : "https://www.minerium.com/" + }, "/Buffett/": { "name" : "Lubian.com", "link" : "" @@ -504,15 +508,15 @@ "name" : "OKKONG", "link" : "https://hash.okkong.com" }, - "/TMSPOOL/" : { - "name" : "TMSPool", + "/AAOPOOL/" : { + "name" : "AAO Pool", "link" : "https://btc.tmspool.top" }, "/one_more_mcd/" : { "name" : "EMCDPool", "link" : "https://pool.emcd.io" }, - "/Foundry USA Pool #dropgold/" : { + "Foundry USA Pool" : { "name" : "Foundry USA", "link" : "https://foundrydigital.com/" }, @@ -539,9 +543,29 @@ "/PureBTC.COM/": { "name": "PureBTC.COM", "link": "https://purebtc.com" + }, + "MARA Pool": { + "name": "MARA Pool", + "link": "https://marapool.com" + }, + "KuCoinPool": { + "name": "KuCoinPool", + "link": "https://www.kucoin.com/mining-pool/" + }, + "Entrustus" : { + "name": "Entrust Charity Pool", + "link": "pool.entustus.org" } }, "payout_addresses" : { + "1MkCDCzHpBsYQivp8MxjY5AkTGG1f2baoe": { + "name": "Luxor", + "link": "https://mining.luxor.tech" + }, + "1ArTPjj6pV3aNRhLPjJVPYoxB98VLBzUmb": { + "name" : "KuCoinPool", + "link" : "https://www.kucoin.com/mining-pool/" + }, "3Bmb9Jig8A5kHdDSxvDZ6eryj3AXd3swuJ": { "name" : "NovaBlock", "link" : "https://novablock.com" @@ -606,7 +630,7 @@ "name" : "BitMinter", "link" : "http://bitminter.com/" }, - "15xiShqUqerfjFdyfgBH1K7Gwp6cbYmsTW " : { + "15xiShqUqerfjFdyfgBH1K7Gwp6cbYmsTW" : { "name" : "EclipseMC", "link" : "https://eclipsemc.com/" }, @@ -634,6 +658,14 @@ "name" : "Huobi.pool", "link" : "https://www.hpt.com/" }, + "1BDbsWi3Mrcjp1wdop3PWFNCNZtu4R7Hjy" : { + "name" : "EMCDPool", + "link" : "https://pool.emcd.io" + }, + "12QVFmJH2b4455YUHkMpEnWLeRY3eJ4Jb5" : { + "name" : "AAO Pool", + "link" : "https://btc.tmspool.top " + }, "1ALA5v7h49QT7WYLcRsxcXqXUqEqaWmkvw" : { "name" : "CloudHashing", "link" : "https://cloudhashing.com/" @@ -915,7 +947,7 @@ "link" : "http://www.dpool.top/" }, "1FbBbv5oYqFKwiPm4CAqvAy8345n8AQ74b" : { - "name" : "Rawpool.com", + "name" : "Rawpool", "link" : "https://www.rawpool.com/" }, "1LsFmhnne74EmU4q4aobfxfrWY4wfMVd8w" : { @@ -934,6 +966,22 @@ "name" : "Poolin", "link" : "https://www.poolin.com/" }, + "1E8CZo2S3CqWg1VZSJNFCTbtT8hZPuQ2kB" : { + "name" : "Poolin", + "link" : "https://www.poolin.com/" + }, + "14sA8jqYQgMRQV9zUtGFvpeMEw7YDn77SK" : { + "name" : "Poolin", + "link" : "https://www.poolin.com/" + }, + "1GNgwA8JfG7Kc8akJ8opdNWJUihqUztfPe" : { + "name" : "Poolin", + "link" : "https://www.poolin.com/" + }, + "17tUZLvy3X2557JGhceXRiij2TNYuhRr4r" : { + "name" : "Poolin", + "link" : "https://www.poolin.com/" + }, "12Taz8FFXQ3E2AGn3ZW1SZM5bLnYGX4xR6" : { "name" : "Tangpool", "link" : "http://www.tangpool.com/" @@ -1126,6 +1174,10 @@ "name" : "Binance Pool", "link" : "https://pool.binance.com/" }, + "1JvXhnHCi6XqcanvrZJ5s2Qiv4tsmm2UMy": { + "name" : "Binance Pool", + "link" : "https://pool.binance.com/" + }, "34Jpa4Eu3ApoPVUKNTN2WeuXVVq1jzxgPi": { "name" : "Lubian.com", "link" : "http://www.lubian.com/" @@ -1173,6 +1225,14 @@ "3CLigLYNkrtoNgNcUwTaKoUSHCwr9W851W": { "name": "Rawpool", "link": "https://www.rawpool.com" + }, + "bc1qf274x7penhcd8hsv3jcmwa5xxzjl2a6pa9pxwm": { + "name" : "F2Pool", + "link" : "https://www.f2pool.com/" + }, + "1A32KFEX7JNPmU1PVjrtiXRrTQcesT3Nf1": { + "name": "MARA Pool", + "link": "https://marapool.com" } } -} +} \ No newline at end of file diff --git a/frontend/cypress/integration/mainnet/mainnet.spec.ts b/frontend/cypress/integration/mainnet/mainnet.spec.ts index 1c6907db0..752617092 100644 --- a/frontend/cypress/integration/mainnet/mainnet.spec.ts +++ b/frontend/cypress/integration/mainnet/mainnet.spec.ts @@ -274,19 +274,6 @@ describe('Mainnet', () => { }); }); }); - - it('loads genesis block and click on the arrow left', () => { - cy.viewport('macbook-16'); - cy.visit('/block/0'); - cy.waitForSkeletonGone(); - cy.waitForPageIdle(); - cy.get('[ngbtooltip="Next Block"] > .ng-fa-icon > .svg-inline--fa').should('be.visible'); - cy.get('[ngbtooltip="Previous Block"] > .ng-fa-icon > .svg-inline--fa').should('not.exist'); - cy.get('[ngbtooltip="Next Block"] > .ng-fa-icon > .svg-inline--fa').click().then(() => { - cy.get('[ngbtooltip="Next Block"] > .ng-fa-icon > .svg-inline--fa').should('be.visible'); - cy.get('[ngbtooltip="Previous Block"] > .ng-fa-icon > .svg-inline--fa').should('be.visible'); - }); - }); }); }); @@ -321,10 +308,10 @@ describe('Mainnet', () => { cy.get(':nth-child(3) > #bitcoin-block-0').should('not.exist'); }); - it('loads the blocks screen', () => { + it('loads the pools screen', () => { cy.visit('/'); cy.waitForSkeletonGone(); - cy.get('#btn-blocks').click().then(() => { + cy.get('#btn-pools').click().then(() => { cy.waitForPageIdle(); }); }); @@ -384,6 +371,112 @@ describe('Mainnet', () => { cy.get('.blockchain-wrapper').should('not.visible'); }); + it('loads genesis block and click on the arrow left', () => { + cy.viewport('macbook-16'); + cy.visit('/block/0'); + cy.waitForSkeletonGone(); + cy.waitForPageIdle(); + cy.get('[ngbtooltip="Next Block"] > .ng-fa-icon > .svg-inline--fa').should('be.visible'); + cy.get('[ngbtooltip="Previous Block"] > .ng-fa-icon > .svg-inline--fa').should('not.exist'); + cy.get('[ngbtooltip="Next Block"] > .ng-fa-icon > .svg-inline--fa').click().then(() => { + cy.get('[ngbtooltip="Next Block"] > .ng-fa-icon > .svg-inline--fa').should('be.visible'); + cy.get('[ngbtooltip="Previous Block"] > .ng-fa-icon > .svg-inline--fa').should('be.visible'); + }); + }); + + it('loads skeleton when changes between networks', () => { + cy.visit('/'); + cy.waitForSkeletonGone(); + + cy.changeNetwork("testnet"); + cy.changeNetwork("signet"); + cy.changeNetwork("mainnet"); + }); + + it.skip('loads the dashboard with the skeleton blocks', () => { + cy.mockMempoolSocket(); + cy.visit("/"); + cy.get(':nth-child(1) > #bitcoin-block-0').should('be.visible'); + cy.get(':nth-child(2) > #bitcoin-block-0').should('be.visible'); + cy.get(':nth-child(3) > #bitcoin-block-0').should('be.visible'); + cy.get('#mempool-block-0').should('be.visible'); + cy.get('#mempool-block-1').should('be.visible'); + cy.get('#mempool-block-2').should('be.visible'); + + emitMempoolInfo({ + 'params': { + command: 'init' + } + }); + + cy.get(':nth-child(1) > #bitcoin-block-0').should('not.exist'); + cy.get(':nth-child(2) > #bitcoin-block-0').should('not.exist'); + cy.get(':nth-child(3) > #bitcoin-block-0').should('not.exist'); + }); + + it('loads the pools screen', () => { + cy.visit('/'); + cy.waitForSkeletonGone(); + cy.get('#btn-pools').click().then(() => { + cy.wait(1000); + }); + }); + + it('loads the graphs screen', () => { + cy.visit('/'); + cy.waitForSkeletonGone(); + cy.get('#btn-graphs').click().then(() => { + cy.wait(1000); + }); + }); + + describe('graphs page', () => { + it('check buttons - mobile', () => { + cy.viewport('iphone-6'); + cy.visit('/graphs'); + cy.waitForSkeletonGone(); + cy.get('.small-buttons > :nth-child(2)').should('be.visible'); + cy.get('#dropdownFees').should('be.visible'); + cy.get('.btn-group').should('be.visible'); + }); + it('check buttons - tablet', () => { + cy.viewport('ipad-2'); + cy.visit('/graphs'); + cy.waitForSkeletonGone(); + cy.get('.small-buttons > :nth-child(2)').should('be.visible'); + cy.get('#dropdownFees').should('be.visible'); + cy.get('.btn-group').should('be.visible'); + }); + it('check buttons - desktop', () => { + cy.viewport('macbook-16'); + cy.visit('/graphs'); + cy.waitForSkeletonGone(); + cy.get('.small-buttons > :nth-child(2)').should('be.visible'); + cy.get('#dropdownFees').should('be.visible'); + cy.get('.btn-group').should('be.visible'); + }); + }); + + it('loads the tv screen - desktop', () => { + cy.viewport('macbook-16'); + cy.visit('/'); + cy.waitForSkeletonGone(); + cy.get('#btn-tv').click().then(() => { + cy.viewport('macbook-16'); + cy.get('.chart-holder'); + cy.get('.blockchain-wrapper').should('be.visible'); + cy.get('#mempool-block-0').should('be.visible'); + }); + }); + + it('loads the tv screen - mobile', () => { + cy.viewport('iphone-6'); + cy.visit('/tv'); + cy.waitForSkeletonGone(); + cy.get('.chart-holder'); + cy.get('.blockchain-wrapper').should('not.visible'); + }); + it('loads the api screen', () => { cy.visit('/'); cy.waitForSkeletonGone(); diff --git a/frontend/cypress/integration/signet/signet.spec.ts b/frontend/cypress/integration/signet/signet.spec.ts index 9ebf67b81..d2bbd1196 100644 --- a/frontend/cypress/integration/signet/signet.spec.ts +++ b/frontend/cypress/integration/signet/signet.spec.ts @@ -44,10 +44,10 @@ describe('Signet', () => { cy.get(':nth-child(3) > #bitcoin-block-0').should('not.exist'); }); - it('loads the blocks screen', () => { + it('loads the pools screen', () => { cy.visit('/signet'); cy.waitForSkeletonGone(); - cy.get('#btn-blocks').click().then(() => { + cy.get('#btn-pools').click().then(() => { cy.wait(1000); }); }); diff --git a/frontend/cypress/integration/testnet/testnet.spec.ts b/frontend/cypress/integration/testnet/testnet.spec.ts index 6f3264244..c0c07aa74 100644 --- a/frontend/cypress/integration/testnet/testnet.spec.ts +++ b/frontend/cypress/integration/testnet/testnet.spec.ts @@ -44,10 +44,10 @@ describe('Testnet', () => { cy.get(':nth-child(3) > #bitcoin-block-0').should('not.exist'); }); - it('loads the blocks screen', () => { + it('loads the pools screen', () => { cy.visit('/testnet'); cy.waitForSkeletonGone(); - cy.get('#btn-blocks').click().then(() => { + cy.get('#btn-pools').click().then(() => { cy.wait(1000); }); }); diff --git a/frontend/server.ts b/frontend/server.ts index af27fcd08..df4ab1294 100644 --- a/frontend/server.ts +++ b/frontend/server.ts @@ -6,7 +6,6 @@ import * as express from 'express'; import * as fs from 'fs'; import * as path from 'path'; import * as domino from 'domino'; -import { createProxyMiddleware } from 'http-proxy-middleware'; import { join } from 'path'; import { AppServerModule } from './src/main.server'; @@ -66,6 +65,7 @@ export function app(locale: string): express.Express { server.get('/mempool-block/*', getLocalizedSSR(indexHtml)); server.get('/address/*', getLocalizedSSR(indexHtml)); server.get('/blocks', getLocalizedSSR(indexHtml)); + server.get('/mining/pools', getLocalizedSSR(indexHtml)); server.get('/graphs', getLocalizedSSR(indexHtml)); server.get('/liquid', getLocalizedSSR(indexHtml)); server.get('/liquid/tx/*', getLocalizedSSR(indexHtml)); @@ -86,6 +86,7 @@ export function app(locale: string): express.Express { server.get('/testnet/mempool-block/*', getLocalizedSSR(indexHtml)); server.get('/testnet/address/*', getLocalizedSSR(indexHtml)); server.get('/testnet/blocks', getLocalizedSSR(indexHtml)); + server.get('/testnet/mining/pools', getLocalizedSSR(indexHtml)); server.get('/testnet/graphs', getLocalizedSSR(indexHtml)); server.get('/testnet/api', getLocalizedSSR(indexHtml)); server.get('/testnet/tv', getLocalizedSSR(indexHtml)); @@ -97,6 +98,7 @@ export function app(locale: string): express.Express { server.get('/signet/mempool-block/*', getLocalizedSSR(indexHtml)); server.get('/signet/address/*', getLocalizedSSR(indexHtml)); server.get('/signet/blocks', getLocalizedSSR(indexHtml)); + server.get('/signet/mining/pools', getLocalizedSSR(indexHtml)); server.get('/signet/graphs', getLocalizedSSR(indexHtml)); server.get('/signet/api', getLocalizedSSR(indexHtml)); server.get('/signet/tv', getLocalizedSSR(indexHtml)); diff --git a/frontend/src/app/app-routing.module.ts b/frontend/src/app/app-routing.module.ts index 43705b85e..36a53781f 100644 --- a/frontend/src/app/app-routing.module.ts +++ b/frontend/src/app/app-routing.module.ts @@ -22,6 +22,7 @@ import { BisqMasterPageComponent } from './components/bisq-master-page/bisq-mast import { SponsorComponent } from './components/sponsor/sponsor.component'; import { LiquidMasterPageComponent } from './components/liquid-master-page/liquid-master-page.component'; import { PushTransactionComponent } from './components/push-transaction/push-transaction.component'; +import { PoolRankingComponent } from './components/pool-ranking/pool-ranking.component'; let routes: Routes = [ { @@ -58,6 +59,10 @@ let routes: Routes = [ path: 'blocks', component: LatestBlocksComponent, }, + { + path: 'mining/pools', + component: PoolRankingComponent, + }, { path: 'graphs', component: StatisticsComponent, @@ -142,6 +147,10 @@ let routes: Routes = [ path: 'blocks', component: LatestBlocksComponent, }, + { + path: 'mining/pools', + component: PoolRankingComponent, + }, { path: 'graphs', component: StatisticsComponent, @@ -220,6 +229,10 @@ let routes: Routes = [ path: 'blocks', component: LatestBlocksComponent, }, + { + path: 'mining/pools', + component: PoolRankingComponent, + }, { path: 'graphs', component: StatisticsComponent, diff --git a/frontend/src/app/app.module.ts b/frontend/src/app/app.module.ts index 3e2c40b25..f9eae0666 100644 --- a/frontend/src/app/app.module.ts +++ b/frontend/src/app/app.module.ts @@ -37,6 +37,7 @@ import { IncomingTransactionsGraphComponent } from './components/incoming-transa import { TimeSpanComponent } from './components/time-span/time-span.component'; import { SeoService } from './services/seo.service'; import { MempoolGraphComponent } from './components/mempool-graph/mempool-graph.component'; +import { PoolRankingComponent } from './components/pool-ranking/pool-ranking.component'; import { LbtcPegsGraphComponent } from './components/lbtc-pegs-graph/lbtc-pegs-graph.component'; import { AssetComponent } from './components/asset/asset.component'; import { AssetsComponent } from './assets/assets.component'; @@ -48,7 +49,7 @@ import { FeesBoxComponent } from './components/fees-box/fees-box.component'; import { DashboardComponent } from './dashboard/dashboard.component'; import { DifficultyComponent } from './components/difficulty/difficulty.component'; import { FontAwesomeModule, FaIconLibrary } from '@fortawesome/angular-fontawesome'; -import { faFilter, faAngleDown, faAngleUp, faAngleRight, faAngleLeft, faBolt, faChartArea, faCogs, faCubes, faDatabase, faExchangeAlt, faInfoCircle, +import { faFilter, faAngleDown, faAngleUp, faAngleRight, faAngleLeft, faBolt, faChartArea, faCogs, faCubes, faHammer, faDatabase, faExchangeAlt, faInfoCircle, faLink, faList, faSearch, faCaretUp, faCaretDown, faTachometerAlt, faThList, faTint, faTv, faAngleDoubleDown, faSortUp, faAngleDoubleUp, faChevronDown, faFileAlt, faRedoAlt, faArrowAltCircleRight, faExternalLinkAlt, faBook, faListUl } from '@fortawesome/free-solid-svg-icons'; import { ApiDocsComponent } from './components/docs/api-docs.component'; import { DocsComponent } from './components/docs/docs.component'; @@ -91,6 +92,7 @@ import { NgbModule } from '@ng-bootstrap/ng-bootstrap'; FeeDistributionGraphComponent, IncomingTransactionsGraphComponent, MempoolGraphComponent, + PoolRankingComponent, LbtcPegsGraphComponent, AssetComponent, AssetsComponent, @@ -143,6 +145,7 @@ export class AppModule { library.addIcons(faTv); library.addIcons(faTachometerAlt); library.addIcons(faCubes); + library.addIcons(faHammer); library.addIcons(faCogs); library.addIcons(faThList); library.addIcons(faList); diff --git a/frontend/src/app/components/about/about.component.html b/frontend/src/app/components/about/about.component.html index 847ed152a..b9050d41e 100644 --- a/frontend/src/app/components/about/about.component.html +++ b/frontend/src/app/components/about/about.component.html @@ -102,7 +102,7 @@ RoninDojo - + Citadel @@ -220,7 +220,7 @@
- Copyright © 2019-2021
+ Copyright © 2019-2022
The Mempool Open Source Project

diff --git a/frontend/src/app/components/difficulty/difficulty.component.html b/frontend/src/app/components/difficulty/difficulty.component.html index 5064c1c08..fc3030286 100644 --- a/frontend/src/app/components/difficulty/difficulty.component.html +++ b/frontend/src/app/components/difficulty/difficulty.component.html @@ -39,13 +39,22 @@ {{ epochData.previousRetarget | absolute | number: '1.2-2' }} %

-
+
Current Period
{{ epochData.progress | number: '1.2-2' }} %
 
+
+
Next halving
+
+ + {{ i }} blocks + {{ i }} block +
+
+
diff --git a/frontend/src/app/components/difficulty/difficulty.component.ts b/frontend/src/app/components/difficulty/difficulty.component.ts index 312c1b2d0..ff44e5aeb 100644 --- a/frontend/src/app/components/difficulty/difficulty.component.ts +++ b/frontend/src/app/components/difficulty/difficulty.component.ts @@ -14,6 +14,8 @@ interface EpochProgress { timeAvg: string; remainingTime: number; previousRetarget: number; + blocksUntilHalving: number; + timeUntilHalving: number; } @Component({ @@ -26,6 +28,9 @@ export class DifficultyComponent implements OnInit { isLoadingWebSocket$: Observable; difficultyEpoch$: Observable; + @Input() showProgress: boolean = true; + @Input() showHalving: boolean = false; + constructor( public stateService: StateService, ) { } @@ -92,6 +97,9 @@ export class DifficultyComponent implements OnInit { colorPreviousAdjustments = '#ffffff66'; } + const blocksUntilHalving = block.height % 210000; + const timeUntilHalving = (blocksUntilHalving * timeAvgMins * 60 * 1000) + (now * 1000); + return { base: `${progress}%`, change, @@ -104,6 +112,8 @@ export class DifficultyComponent implements OnInit { newDifficultyHeight, remainingTime, previousRetarget, + blocksUntilHalving, + timeUntilHalving, }; }) ); diff --git a/frontend/src/app/components/master-page/master-page.component.html b/frontend/src/app/components/master-page/master-page.component.html index f05b297c7..4624340d3 100644 --- a/frontend/src/app/components/master-page/master-page.component.html +++ b/frontend/src/app/components/master-page/master-page.component.html @@ -31,8 +31,8 @@ -