mirrors/blockstream-satellite-api
1
0
Fork 0
mirror of https://github.com/Blockstream/satellite-api.git synced 2024-11-19 04:50:01 +01:00
Code Issues Packages Projects Releases Wiki Activity

Support multiple parallel logical message channels

Browse Source 
The same server now can handle multiple logical channels, on which the
transmitter logic runs independently. That is, while previously a single
message would be in transmitting state at a time, now multiple messages
can be in transmitting state as long as they belong to distinct logical
channels.

The supported channels each have different permissions. The user channel
is where users can post, get, and delete messages as needed. In
contrast, the other channels do not grant all permissions to users. Some
are read-only (users can get but not post) and there is a channel (the
auth channel) on which users have no permissions (neither get nor post).

For the channels on which users do not have all permissions (get, post,
and delete), this patch adds admin-specific routes, which are prefixed
by /admin/. The /admin/ route is protected via SSL in production and
allows the admin host to send GET/POST/DELETE requests normally. Hence,
for instance, the admin host can post a message on the auth channel
(with POST /admin/order) and read it (with GET /admin/order) for
transmission over satellite, whereas regulars cannot. With this scheme,
the auth channel messages are accessible exclusively over satellite (and
not over the internet).

The admin routes were added to the following endpoints:
- /order/<uuid> (GET and DELETE requests)
- /order (POST request)
- /orders/<state> (GET request)
- /message/<tx_seq_num> (GET request)

The messages posted by the admin are not paid, so this patch removes the
requirement of invoice generation and payment. Only paid orders now
generate an invoice. Thus, the POST request to the /order/ endpoint does
not return an invoice for non-paid (admin-only) messages.

Also, this patch updates the queue page to display the orders separately
for each channel. The query string channel parameter determines which
channel the page shows.

Finally, this patch updates the events published into the Redis db on
transmission. The event includes the corresponding logical channel so
that SSE events can be subscribed independently for each channel.
This commit is contained in:
Blockstream Satellite 2023-01-31 18:14:55 -03:00
parent f7695da16c
commit 50df236d6b
19 changed files with 661 additions and 104 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
README.md
View File

@ -5,25 +5,24 @@
A lightning app (Lapp) based on c-lightning. Presents an API to submit messages for global broadcast over Blockstream Satellite with payments via Bitcoin Lightning.
<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-generate-toc again -->
## Contents
- [Setup](#setup)
- [Run](#run)
- [Example Applications](#example-applications)
- [REST API](#rest-api)
- [Satellite API](#satellite-api)
- [Setup](#setup)
- [Run](#run)
- [Example Applications](#example-applications)
- [REST API](#rest-api)
- [POST /order](#post-order)
- [POST /order/:uuid/bump](#post-orderuuidbump)
- [GET /order/:uuid](#get-orderuuid)
- [GET /order/:uuid/sent_message](#get-orderuuidsentmessage)
- [DELETE /order/:uuid](#delete-orderuuid)
- [GET /orders/pending](#get-orderspending)
- [GET /orders/queued](#get-ordersqueued)
- [GET /orders/sent](#get-orderssent)
- [GET /message/:seq\_num](#get-messageseq_num)
- [GET /info](#get-info)
- [GET /subscribe/:channels](#get-subscribechannels)
- [Debugging](#debugging)
- [Queue Page](#queue-page)
- [Future Work](#future-work)
- [Future Work](#future-work)
<!-- markdown-toc end -->
@ -73,7 +72,7 @@ If successful, the response includes the JSON Lightning invoice as returned by L
{"auth_token":"d784e322dad7ec2671086ce3ad94e05108f2501180d8228577fbec4115774750","uuid":"409348bc-6af0-4999-b715-4136753979df","lightning_invoice":{"id":"N0LOTYc9j0gWtQVjVW7pK","msatoshi":"514200","description":"BSS Test","rhash":"5e5c9d111bc76ce4bf9b211f12ca2d9b66b81ae9839b4e530b16cedbef653a3a","payreq":"lntb5142n1pd78922pp5tewf6ygmcakwf0umyy039j3dndntsxhfswd5u5ctzm8dhmm98gaqdqdgff4xgz5v4ehgxqzjccqp286gfgrcpvzl04sdg2f9sany7ptc5aracnd6kvr2nr0e0x5ajpmfhsjkqzw679ytqgnt6w4490jjrgcvuemz790salqyz9far68cpqtgq3q23el","expires_at":1541642146,"created_at":1541641546,"metadata":{"sha256_message_digest":"0e2bddf3bba1893b5eef660295ef12d6fc72870da539c328cf24e9e6dbb00f00","uuid":"409348bc-6af0-4999-b715-4136753979df"},"status":"unpaid"}}
```
Error codes that can be returned by this endpoint include: `BID_TOO_SMALL` (102), `MESSAGE_FILE_TOO_SMALL` (117), `MESSAGE_FILE_TOO_LARGE` (118), `MESSAGE_MISSING` (126).
The error codes that can be returned by this endpoint include `BID_TOO_SMALL` (102), `MESSAGE_FILE_TOO_SMALL` (117), `MESSAGE_FILE_TOO_LARGE` (118), `MESSAGE_MISSING` (126), and `ORDER_CHANNEL_UNAUTHORIZED_OP` (130).
### POST /order/:uuid/bump ###
@ -87,7 +86,7 @@ Response object is in the same format as for `POST /order`.
As shown below for DELETE, the `auth_token` may alternatively be provided using the `X-Auth-Token` HTTP header.
Error codes that can be returned by this endpoint include: `INVALID_AUTH_TOKEN` (109), `ORDER_NOT_FOUND` (104).
The error codes that can be returned by this endpoint include `INVALID_AUTH_TOKEN` (109), `ORDER_NOT_FOUND` (104), and `ORDER_CHANNEL_UNAUTHORIZED_OP` (130).
### GET /order/:uuid ###
@ -97,7 +96,7 @@ Retrieve an order by UUID. Must provide the corresponding auth token to prove th
curl -v -H "X-Auth-Token: 5248b13a722cd9b2e17ed3a2da8f7ac6bd9a8fe7130357615e074596e3d5872f" $SATELLITE_API/order/409348bc-6af0-4999-b715-4136753979df
```
Error codes that can be returned by this endpoint include: `INVALID_AUTH_TOKEN` (109), `ORDER_NOT_FOUND` (104).
The error codes that can be returned by this endpoint include `INVALID_AUTH_TOKEN` (109), `ORDER_NOT_FOUND` (104), and `ORDER_CHANNEL_UNAUTHORIZED_OP` (130).
### DELETE /order/:uuid ###
@ -160,6 +159,16 @@ curl $SATELLITE_API/orders/sent?before=2019-01-16T18:13:46-08:00
The response is a JSON array of records (one for each queued message). The revealed fields for each record include: `uuid`, `bid`, `bid_per_byte`, `message_size`, `message_digest`, `status`, `created_at`, `started_transmission_at`, and `ended_transmission_at`.
### GET /message/:seq_num
Retrieve a transmitted message by its unique sequence number. For example:
```bash
curl -v $SATELLITE_API/message/3
```
The error codes that can be returned by this endpoint include `SEQUENCE_NUMBER_NOT_FOUND` (114) and `ORDER_CHANNEL_UNAUTHORIZED_OP` (130).
### GET /info
Returns information about the c-lightning node where satellite API payments are terminated. The response is a JSON object consisting of the node ID, port, IP addresses, and other information useful for opening payment channels. For example:
@ -170,16 +179,12 @@ Returns information about the c-lightning node where satellite API payments are
### GET /subscribe/:channels
Subscribe to one or more [server-sent events](https://en.wikipedia.org/wiki/Server-sent_events) channels. The `channels` parameter is a comma-separated list of event channels. Currently, only one channel is available: `transmissions`, to which an event is pushed each time a message transmission begins and ends. Event data includes a JSON representation of the order, including its current status.
Subscribe to one or more [server-sent events](https://en.wikipedia.org/wiki/Server-sent_events) channels. The `channels` parameter is a comma-separated list of event channels. Currently, the following channels are available: `transmissions`, `auth`, `gossip`, and `btc-src`. An event is broadcast on a channel each time a message transmission begins and ends on that channel. The event data consists of the order's JSON representation, including its current status.
```bash
curl $SATELLITE_API/subscribe/:channels
```
Error codes that can be returned by this endpoint include: `CHANNELS_EQUALITY` (124).
## Debugging ##
### Queue Page ###
A simple table view of queued, pending and sent messages is available at `$SATELLITE_API/queue.html`. This page can be used for debugging and as an example for building a web front-end to the satellite API.

2
docker-compose.yml
View File

@ -42,7 +42,7 @@ services:
links:
- redis
environment:
- SUB_CHANNELS=transmissions
- SUB_CHANNELS=transmissions,gossip,btc-src,auth
- REDIS_URI=redis://redis:6379
redis:
image: "redis:latest"

25
server/alembic/versions/3ec897840ea4_add_channel_to_order_table.py Normal file
View File

@ -0,0 +1,25 @@
"""Add channel to order table
Revision ID: 3ec897840ea4
Revises: 0704901102eb
Create Date: 2022-04-05 21:57:27.398804
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers, used by Alembic.
revision = '3ec897840ea4'
down_revision = '0704901102eb'
branch_labels = None
depends_on = None
def upgrade():
op.add_column(
'orders',
sa.Column('channel', sa.Integer, default=1, server_default='1'))
def downgrade():
op.drop_column('orders', 'channel')

39
server/constants.py
View File

@ -68,4 +68,41 @@ TRANSMIT_RATE = int(os.getenv('TRANSMIT_RATE',
LOGGING_FORMAT = '%(asctime)s %(levelname)s %(name)s : %(message)s'
REDIS_URI = os.getenv('REDIS_URI', 'redis://127.0.0.1:6379')
SUB_CHANNELS = ['transmissions']
USER_CHANNEL = 1
AUTH_CHANNEL = 3
GOSSIP_CHANNEL = 4
BTC_SRC_CHANNEL = 5
class ChannelInfo:
def __init__(self, name, user_permissions):
"""Construct channel information
Args:
name (str): Channel name.
user_permissions (list): User permissions. An empty list means the
channel messages are only sent over satellite. A list with
'get' permission only means the users can only fetch messages
but not post them, and only the admin can post messages.
"""
assert isinstance(user_permissions, list)
assert len(user_permissions) == 0 or \
[x in ['get', 'post'] for x in user_permissions]
self.name = name
self.user_permissions = user_permissions
# Attribute indicating whether the channel messages must be paid by the
# user. The channels on which users can post messages necessarily
# require payment. The other channels can only have messages posted by
# the admin, and these messages are not paid.
self.requires_payment = 'post' in user_permissions
CHANNEL_INFO = {
USER_CHANNEL: ChannelInfo('transmissions', ['get', 'post', 'delete']),
AUTH_CHANNEL: ChannelInfo('auth', []),
GOSSIP_CHANNEL: ChannelInfo('gossip', ['get']),
BTC_SRC_CHANNEL: ChannelInfo('btc-src', ['get']),
}
CHANNELS = list(CHANNEL_INFO.keys())

11
server/error.py
View File

@ -36,9 +36,6 @@ errors = {
HTTPStatus.NOT_FOUND),
'INVOICE_ALREADY_PAID': (123, "Payment problem", "Invoice already paid",
HTTPStatus.BAD_REQUEST),
'CHANNELS_EQUALITY': (124, "invalid channel",
"channel {} is not a valid channel name",
HTTPStatus.INTERNAL_SERVER_ERROR),
'MESSAGE_MISSING':
(126, "Message upload problem",
"Either a message file or a message parameter is required",
@ -47,8 +44,12 @@ errors = {
(128, "Lightning Charge communication error",
"Failed to fetch information about the Lightning node",
HTTPStatus.INTERNAL_SERVER_ERROR),
'INVOICE_ALREADY_EXPIRED':
(129, "Payment problem", "Invoice already expired", HTTPStatus.BAD_REQUEST)
'INVOICE_ALREADY_EXPIRED': (129, "Payment problem",
"Invoice already expired",
HTTPStatus.BAD_REQUEST),
'ORDER_CHANNEL_UNAUTHORIZED_OP': (130, "Unauthorized channel operation",
"Operation not supported on channel {}",
HTTPStatus.UNAUTHORIZED),
}

2
server/invoices.py
View File

@ -24,6 +24,6 @@ class InvoiceResource(Resource):
order = Order.query.filter_by(id=invoice.order_id).first()
if order.status == OrderStatus.paid.value:
transmitter.tx_start()
transmitter.tx_start(order.channel)
return {'message': f'invoice {invoice.lid} paid'}

1
server/models.py
View File

@ -19,6 +19,7 @@ class Order(db.Model):
tx_seq_num = db.Column(db.Integer, unique=True)
unpaid_bid = db.Column(db.Integer, nullable=False)
region_code = db.Column(db.Integer)
channel = db.Column(db.Integer, default=1)
invoices = db.relationship('Invoice', backref='order', lazy=True)

7
server/order_helpers.py
View File

@ -402,13 +402,14 @@ def refresh_retransmission_table():
upsert_retransmission(order)
def get_next_retransmission():
def get_next_retransmission(channel):
"""Get the next highest bidding order requiring retransmission"""
refresh_retransmission_table()
orders_with_retry_info = db.session.query(
Order, TxRetry).filter(Order.id == TxRetry.order_id).order_by(
Order.bid_per_byte.desc()).all()
Order, TxRetry).filter(Order.id == TxRetry.order_id).filter(
Order.channel == channel).order_by(
Order.bid_per_byte.desc()).all()
for order, retry_info in orders_with_retry_info:
if retry_info.pending:

113
server/orders.py
View File

@ -6,12 +6,11 @@ import os
from uuid import uuid4
from flask import current_app, request, send_file
from flask_restful import Resource
from marshmallow import ValidationError
from sqlalchemy import or_
from sqlalchemy import and_, or_
from constants import OrderStatus
from constants import CHANNEL_INFO, OrderStatus
from database import db
from error import get_http_error_resp
from invoice_helpers import new_invoice, pay_invoice
@ -39,20 +38,35 @@ def sha256_checksum(filename, block_size=SHA256_BLOCK_SIZE):
class OrderResource(Resource):
def get(self, uuid):
admin_mode = request.path.startswith("/admin/")
success, order_or_error = order_helpers.get_and_authenticate_order(
uuid, request.form, request.args)
if not success:
return order_or_error
order = order_or_error
if not admin_mode and 'get' not in \
constants.CHANNEL_INFO[order.channel].user_permissions:
return get_http_error_resp('ORDER_CHANNEL_UNAUTHORIZED_OP',
order.channel)
return order_schema.dump(order)
def delete(self, uuid):
admin_mode = request.path.startswith("/admin/")
success, order_or_error = order_helpers.get_and_authenticate_order(
uuid, request.form, request.args)
if not success:
return order_or_error
order = order_or_error
if not admin_mode and 'delete' not in \
constants.CHANNEL_INFO[order.channel].user_permissions:
return get_http_error_resp('ORDER_CHANNEL_UNAUTHORIZED_OP',
order.channel)
if order.status != OrderStatus.pending.value and\
order.status != OrderStatus.paid.value:
return get_http_error_resp('ORDER_CANCELLATION_ERROR',
@ -70,15 +84,23 @@ class OrderResource(Resource):
class OrderUploadResource(Resource):
def post(self):
args = request.form
errors = order_upload_req_schema.validate(args)
admin_mode = request.path.startswith("/admin/")
if errors:
return errors, HTTPStatus.BAD_REQUEST
try:
args = order_upload_req_schema.load(request.form)
except ValidationError as error:
return error.messages, HTTPStatus.BAD_REQUEST
has_msg = 'message' in args
has_file = 'file' in request.files
channel = args['channel']
if not admin_mode and 'post' not in \
constants.CHANNEL_INFO[channel].user_permissions:
return get_http_error_resp('ORDER_CHANNEL_UNAUTHORIZED_OP',
channel)
requires_payment = CHANNEL_INFO[channel].requires_payment
if (has_msg and has_file):
return "Choose message or file", HTTPStatus.BAD_REQUEST
@ -87,7 +109,6 @@ class OrderUploadResource(Resource):
uuid = str(uuid4())
filepath = os.path.join(constants.MSG_STORE_PATH, uuid)
bid = int(args.get('bid'))
if (has_msg):
with open(filepath, 'w') as fd:
@ -108,23 +129,28 @@ class OrderUploadResource(Resource):
return get_http_error_resp('MESSAGE_FILE_TOO_LARGE',
constants.MAX_MESSAGE_SIZE / (2**20))
if (not bidding.validate_bid(msg_size, bid)):
bid = int(args.get('bid')) if requires_payment else 0
if (requires_payment and not bidding.validate_bid(msg_size, bid)):
os.remove(filepath)
min_bid = bidding.get_min_bid(msg_size)
return get_http_error_resp('BID_TOO_SMALL', min_bid)
msg_digest = sha256_checksum(filepath)
starting_state = OrderStatus.pending.value if requires_payment \
else OrderStatus.paid.value
new_order = Order(uuid=uuid,
unpaid_bid=bid,
message_size=msg_size,
message_digest=msg_digest,
status=OrderStatus.pending.value)
status=starting_state,
channel=channel)
success, invoice = new_invoice(new_order, bid)
if not success:
return invoice
if requires_payment:
success, invoice = new_invoice(new_order, bid)
if not success:
return invoice
new_order.invoices.append(invoice)
new_order.invoices.append(invoice)
if 'regions' in args:
regions_in_request = json.loads(args['regions'])
new_order.region_code = region_number_list_to_code(
@ -133,28 +159,32 @@ class OrderUploadResource(Resource):
db.session.add(new_order)
db.session.commit()
if constants.FORCE_PAYMENT:
if constants.FORCE_PAYMENT and requires_payment:
current_app.logger.info('force payment of the invoice')
# Force the sequence of actions executed by the invoice callback
pay_invoice(invoice)
transmitter.tx_start()
transmitter.tx_start(new_order.channel)
elif not requires_payment:
transmitter.tx_start(new_order.channel)
return {
# Return the invoice only if the channel requires payment for orders
resp = {
'auth_token': order_helpers.compute_auth_token(uuid),
'uuid': uuid,
'lightning_invoice': json.loads(invoice.invoice)
'uuid': uuid
}
if requires_payment:
resp['lightning_invoice'] = json.loads(invoice.invoice)
return resp
class BumpOrderResource(Resource):
def post(self, uuid):
query_args = request.args
form_args = request.form
errors = order_bump_schema.validate(form_args)
if errors:
return errors, HTTPStatus.BAD_REQUEST
try:
form_args = order_bump_schema.load(request.form)
except ValidationError as error:
return error.messages, HTTPStatus.BAD_REQUEST
success, order_or_error = order_helpers.get_and_authenticate_order(
uuid, form_args, query_args)
@ -162,6 +192,10 @@ class BumpOrderResource(Resource):
return order_or_error
order = order_or_error
if not CHANNEL_INFO[order.channel].requires_payment:
return get_http_error_resp('ORDER_CHANNEL_UNAUTHORIZED_OP',
order.channel)
if order.status != OrderStatus.pending.value and\
order.status != OrderStatus.paid.value:
return get_http_error_resp('ORDER_BUMP_ERROR',
@ -185,6 +219,7 @@ class BumpOrderResource(Resource):
class OrdersResource(Resource):
def get(self, state):
admin_mode = request.path.startswith("/admin/")
if state not in ['pending', 'queued', 'sent']:
return {
state: [
@ -200,29 +235,36 @@ class OrdersResource(Resource):
before = db.func.datetime(args['before'])
limit = args['limit']
channel = args['channel']
if not admin_mode and 'get' not in \
constants.CHANNEL_INFO[channel].user_permissions:
return get_http_error_resp('ORDER_CHANNEL_UNAUTHORIZED_OP',
channel)
if state == 'pending':
orders = Order.query.filter_by(
status=OrderStatus[state].value).\
orders = Order.query.filter(and_(
Order.channel == channel,
Order.status == OrderStatus[state].value)).\
filter(db.func.datetime(Order.created_at) < before).\
order_by(Order.created_at.desc()).\
limit(limit)
elif state == 'queued':
orders = Order.query.filter(or_(
orders = Order.query.filter(and_(Order.channel == channel, or_(
Order.status ==
OrderStatus.transmitting.value,
Order.status ==
OrderStatus.confirming.value,
Order.status ==
OrderStatus.paid.value)).\
OrderStatus.paid.value))).\
filter(db.func.datetime(Order.created_at) < before).\
order_by(Order.bid_per_byte.desc()).limit(limit)
elif state == 'sent':
orders = Order.query.filter(or_(
orders = Order.query.filter(and_(Order.channel == channel, or_(
Order.status ==
OrderStatus.sent.value,
Order.status ==
OrderStatus.received.value)).\
OrderStatus.received.value))).\
filter(db.func.datetime(Order.created_at) < before).\
order_by(Order.ended_transmission_at.desc()).\
limit(limit)
@ -250,6 +292,8 @@ class GetMessageResource(Resource):
class GetMessageBySeqNumResource(Resource):
def get(self, tx_seq_num):
admin_mode = request.path.startswith("/admin/")
order = Order.query.filter_by(tx_seq_num=tx_seq_num).filter(
or_(Order.status == OrderStatus.sent.value,
Order.status == OrderStatus.transmitting.value,
@ -258,6 +302,11 @@ class GetMessageBySeqNumResource(Resource):
if not order:
return get_http_error_resp('SEQUENCE_NUMBER_NOT_FOUND', tx_seq_num)
if not admin_mode and 'get' not in \
constants.CHANNEL_INFO[order.channel].user_permissions:
return get_http_error_resp('ORDER_CHANNEL_UNAUTHORIZED_OP',
order.channel)
message_path = os.path.join(constants.MSG_STORE_PATH, order.uuid)
return send_file(message_path,
mimetype='application/json',
@ -317,7 +366,7 @@ class TxConfirmationResource(Resource):
db.session.refresh(order)
if order.status == OrderStatus.confirming.value and \
last_status == OrderStatus.transmitting.value:
transmitter.tx_start()
transmitter.tx_start(order.channel)
return {
'message': f'transmission confirmed for regions {args["regions"]}'

6
server/schemas.py
View File

@ -44,11 +44,13 @@ def must_be_region_number_list(data):
class OrderUploadReqSchema(Schema):
bid = fields.Int(required=True, validate=validate.Range(min=0))
bid = fields.Int(missing=0, validate=validate.Range(min=0))
message = fields.Str(validate=validate.Length(
max=constants.MAX_TEXT_MSG_LEN))
regions = fields.String(required=False,
validate=must_be_region_number_list)
channel = fields.Int(missing=constants.USER_CHANNEL,
validate=validate.OneOf(constants.CHANNELS))
class OrderBumpSchema(Schema):
@ -66,6 +68,8 @@ class OrdersSchema(Schema):
limit = fields.Int(missing=lambda: constants.PAGE_SIZE,
validate=validate.Range(min=1,
max=constants.MAX_PAGE_SIZE))
channel = fields.Int(missing=constants.USER_CHANNEL,
validate=validate.OneOf(constants.CHANNELS))
class TxConfirmationSchema(Schema):

22
server/server.py
View File

@ -10,9 +10,15 @@ import constants
from database import db
from info import InfoResource
from invoices import InvoiceResource
from orders import BumpOrderResource, GetMessageResource,\
GetMessageBySeqNumResource, OrderResource, OrdersResource,\
OrderUploadResource, RxConfirmationResource, TxConfirmationResource
from orders import \
BumpOrderResource,\
GetMessageBySeqNumResource,\
GetMessageResource,\
OrderResource,\
OrdersResource,\
OrderUploadResource,\
RxConfirmationResource,\
TxConfirmationResource
from queues import QueueResource
@ -30,15 +36,17 @@ def create_app(from_test=False):
with app.app_context():
db.create_all()
api = Api(app)
api.add_resource(OrderUploadResource, '/order')
api.add_resource(OrdersResource, '/orders/<state>')
api.add_resource(OrderResource, '/order/<uuid>')
api.add_resource(OrderUploadResource, '/order', '/admin/order')
api.add_resource(OrdersResource, '/orders/<state>',
'/admin/orders/<state>')
api.add_resource(OrderResource, '/order/<uuid>', '/admin/order/<uuid>')
api.add_resource(BumpOrderResource, '/order/<uuid>/bump')
api.add_resource(TxConfirmationResource, '/order/tx/<tx_seq_num>')
api.add_resource(RxConfirmationResource, '/order/rx/<tx_seq_num>')
api.add_resource(InfoResource, '/info')
api.add_resource(InvoiceResource, '/callback/<lid>/<charged_auth_token>')
api.add_resource(GetMessageBySeqNumResource, '/message/<tx_seq_num>')
api.add_resource(GetMessageBySeqNumResource, '/message/<tx_seq_num>',
'/admin/message/<tx_seq_num>')
api.add_resource(QueueResource, '/queue.html')
if constants.env == 'development' or constants.env == 'test':

12
server/templates/queue.html
View File

@ -108,7 +108,13 @@
<script type="text/javascript">
$(function() {
$.getJSON( "orders/queued", function( data ) {
const urlParams = new URLSearchParams(window.location.search);
channel = urlParams.get('channel');
if (!channel) {
channel = 1;
}
$.getJSON( "orders/queued", {'channel': channel}, function( data ) {
$.each( data, function( key, val ) {
var started_transmission_at = moment(moment.utc(val.started_transmission_at).toDate()).local();
var created_at = moment(moment.utc(val.created_at).toDate()).local().fromNow();
@ -124,7 +130,7 @@
});
});
$.getJSON( "orders/pending", function( data ) {
$.getJSON( "orders/pending", {'channel': channel}, function( data ) {
$.each( data, function( key, val ) {
var created_at = moment(moment.utc(val.created_at).toDate()).local().fromNow();
$("#pending_table").append($('<tr>').append(
@ -138,7 +144,7 @@
});
});
$.getJSON( "orders/sent", function( data ) {
$.getJSON( "orders/sent", {'channel': channel}, function( data ) {
let env = '{{ env }}';
$.each( data, function( key, val ) {
var started_transmission_at = moment(moment.utc(val.started_transmission_at).toDate()).local();

24
server/tests/common.py
View File

@ -20,22 +20,30 @@ def rnd_string(n_bytes):
for _ in range(n_bytes))
def upload_test_file(client, msg, bid, regions=[]):
def upload_test_file(client, msg, bid, regions=[], channel=None, admin=False):
post_data = {'bid': bid, 'file': (io.BytesIO(msg.encode()), 'testfile')}
if len(regions) > 0:
post_data['regions'] = [regions]
if channel:
post_data['channel'] = channel
endpoint = '/admin/order' if admin else '/order'
return client.post('/order',
return client.post(endpoint,
data=post_data,
content_type='multipart/form-data')
def place_order(client, n_bytes, regions=[], bid=None):
def place_order(client,
n_bytes,
regions=[],
bid=None,
channel=None,
admin=False):
if bid is None:
bid = bidding.get_min_bid(n_bytes)
msg = rnd_string(n_bytes)
return upload_test_file(client, msg, bid, regions)
return upload_test_file(client, msg, bid, regions, channel, admin)
def check_upload(order_uuid, expected_data):
@ -119,7 +127,9 @@ def generate_test_order(mock_new_invoice,
order_id=1,
regions=[],
confirmed_tx=[],
started_transmission_at=None):
started_transmission_at=None,
channel=None,
admin=False):
"""Generate a valid order and add it to the database
This function generates an order with a related invoice with
@ -144,6 +154,8 @@ def generate_test_order(mock_new_invoice,
regions: list of regions over which this order should be
transmitted. The default value is an empty list implying
the order should be sent over all regions.
channel: Logical channel on which to transmit the order.
admin: Whether to post the order via the /admin/order route.
Returns:
The json response of the create order endpoint.
@ -157,7 +169,7 @@ def generate_test_order(mock_new_invoice,
mock_new_invoice.return_value = (True,
new_invoice(order_id, invoice_status,
bid))
post_rv = place_order(client, n_bytes, regions, bid)
post_rv = place_order(client, n_bytes, regions, bid, channel, admin)
assert post_rv.status_code == HTTPStatus.OK
uuid = post_rv.get_json()['uuid']
# Set order's sequence number and status

192
server/tests/test_order.py
View File

@ -99,6 +99,21 @@ def test_uploaded_text_msg_too_large(client):
assert rv.status_code == HTTPStatus.BAD_REQUEST
def test_post_order_invalid_channel(client):
n_bytes = 10
bid = bidding.get_min_bid(n_bytes)
msg = rnd_string(n_bytes)
rv = client.post('/order',
data={
'bid': bid,
'message': msg.encode(),
'channel': 10
})
assert rv.status_code == HTTPStatus.BAD_REQUEST
assert 'channel' in rv.get_json()
assert "Must be one of" in rv.get_json()['channel'][0]
@patch('orders.new_invoice')
def test_uploaded_text_msg_max_size(mock_new_invoice, client):
n_bytes = constants.MAX_TEXT_MSG_LEN
@ -177,9 +192,9 @@ def test_invoice_generation_failure(mock_new_invoice, client):
def test_get_order(mock_new_invoice, client):
json_response = generate_test_order(mock_new_invoice, client)
uuid = json_response['uuid']
auth_token = json_response['auth_token']
get_rv = client.get(f'/order/{uuid}',
headers={'X-Auth-Token': json_response['auth_token']})
get_rv = client.get(f'/order/{uuid}', headers={'X-Auth-Token': auth_token})
get_json_resp = get_rv.get_json()
assert get_rv.status_code == HTTPStatus.OK
assert get_json_resp['uuid'] == uuid
@ -189,9 +204,9 @@ def test_get_order(mock_new_invoice, client):
def test_get_order_auth_token_as_form_param(mock_new_invoice, client):
json_response = generate_test_order(mock_new_invoice, client)
uuid = json_response['uuid']
auth_token = json_response['auth_token']
get_rv = client.get(f'/order/{uuid}',
data={'auth_token': json_response['auth_token']})
get_rv = client.get(f'/order/{uuid}', data={'auth_token': auth_token})
get_json_resp = get_rv.get_json()
assert get_rv.status_code == HTTPStatus.OK
assert get_json_resp['uuid'] == uuid
@ -201,8 +216,8 @@ def test_get_order_auth_token_as_form_param(mock_new_invoice, client):
def test_get_order_auth_token_as_query_param(mock_new_invoice, client):
json_response = generate_test_order(mock_new_invoice, client)
uuid = json_response['uuid']
auth_token = json_response['auth_token']
get_rv = client.get(f'/order/{uuid}?auth_token={auth_token}')
get_json_resp = get_rv.get_json()
assert get_rv.status_code == HTTPStatus.OK
@ -236,6 +251,29 @@ def test_get_order_missing_auth_token(mock_new_invoice, client):
assert get_rv.status_code == HTTPStatus.UNAUTHORIZED
@patch('orders.new_invoice')
def test_get_admin_order(mock_new_invoice, client):
# Post order on a channel that forbids users from fetching messages
json_response = generate_test_order(mock_new_invoice,
client,
channel=constants.AUTH_CHANNEL,
admin=True)
uuid = json_response['uuid']
auth_token = json_response['auth_token']
# Getting through the normal route should fail
get_rv = client.get(f'/order/{uuid}', headers={'X-Auth-Token': auth_token})
assert get_rv.status_code == HTTPStatus.UNAUTHORIZED
assert_error(get_rv.get_json(), 'ORDER_CHANNEL_UNAUTHORIZED_OP')
# Getting through the admin route should work
get_rv = client.get(f'/admin/order/{uuid}',
headers={'X-Auth-Token': auth_token})
get_json_resp = get_rv.get_json()
assert get_rv.status_code == HTTPStatus.OK
assert get_json_resp['uuid'] == uuid
def test_adjust_bids(client):
# if the values like bid, unpaid_bid, bid_per_byte are wrong or become
# obsolete due to changes in the invoices, the adjust_bids function should
@ -378,6 +416,29 @@ def test_bump_transmitted_order(mock_new_invoice, client):
assert_error(bump_rv.get_json(), 'ORDER_BUMP_ERROR')
@patch('orders.new_invoice')
def test_bump_non_paid_order(mock_new_invoice, client):
# Send order over the gossip channel, which is not paid.
initial_bid = 1000
json_response = generate_test_order(mock_new_invoice,
client,
bid=initial_bid,
channel=constants.GOSSIP_CHANNEL,
admin=True)
uuid = json_response['uuid']
auth_token = json_response['auth_token']
# Since the order is not paid, a bump request should not be authorized
bid_increase = 2500
bump_rv = client.post(f'/order/{uuid}/bump',
data={
'bid_increase': bid_increase,
},
headers={'X-Auth-Token': auth_token})
assert bump_rv.status_code == HTTPStatus.UNAUTHORIZED
assert_error(bump_rv.get_json(), 'ORDER_CHANNEL_UNAUTHORIZED_OP')
@patch('orders.new_invoice')
def test_cancel_order(mock_new_invoice, client):
bid = 1000
@ -440,6 +501,37 @@ def test_cancel_non_existing_order(client):
assert_error(delete_rv.get_json(), 'ORDER_NOT_FOUND')
@patch('orders.new_invoice')
def test_cancel_order_unauthorized_channel_op(mock_new_invoice, client):
# Place order on a channel that forbids uses from deleting orders.
bid = 1000
json_response = generate_test_order(mock_new_invoice,
client,
bid=bid,
channel=constants.GOSSIP_CHANNEL,
admin=True)
uuid = json_response['uuid']
auth_token = json_response['auth_token']
# Deleting through the regular user endpoint should fail
delete_rv = client.delete(f'/order/{uuid}',
headers={'X-Auth-Token': auth_token})
assert delete_rv.status_code == HTTPStatus.UNAUTHORIZED
assert_error(delete_rv.get_json(), 'ORDER_CHANNEL_UNAUTHORIZED_OP')
# Deleting through the admin endpoint should work (i.e., won't return
# unauthorized operation)
delete_rv = client.delete(f'/admin/order/{uuid}',
headers={'X-Auth-Token': auth_token})
# In this case, it hits a cancellation error because the order is already
# in transmitting state, given that the gossip channel is auto-paid.
assert delete_rv.status_code == HTTPStatus.BAD_REQUEST
assert_error(delete_rv.get_json(), 'ORDER_CANCELLATION_ERROR')
db_order = Order.query.filter_by(uuid=uuid).first()
assert db_order.status == OrderStatus.transmitting.value
assert db_order.cancelled_at is None
def test_get_sent_message_for_nonexisting_uuid(client):
# Try to get message for a non existing uuid
rv = client.get('/order/some-uuid/sent_message')
@ -510,6 +602,90 @@ def test_get_sent_message_by_seq_number(mock_new_invoice, client, status):
check_received_message(uuid, received_message)
@patch('orders.new_invoice')
def test_get_sent_message_by_seq_number_unauthorized_channel_op(
mock_new_invoice, client):
# Create an order on the auth channel, which forbids GET requests from
# users. Make sure those requests fail. And use the /admin/order endpoint
# when POSTing the order.
uuid = generate_test_order(mock_new_invoice,
client,
order_status=OrderStatus.sent,
tx_seq_num=1,
channel=constants.AUTH_CHANNEL,
admin=True)['uuid']
# Reading by sequence number via the regular route should fail
rv = client.get('/message/1')
assert rv.status_code == HTTPStatus.UNAUTHORIZED
assert_error(rv.get_json(), 'ORDER_CHANNEL_UNAUTHORIZED_OP')
# Reading by sequence number via the admin route should fail
rv = client.get('/admin/message/1')
assert rv.status_code == HTTPStatus.OK
received_message = rv.data
check_received_message(uuid, received_message)
@patch('orders.new_invoice')
@pytest.mark.parametrize("channel", constants.CHANNELS)
def test_get_sent_message_admin(mock_new_invoice, client, channel):
# the admin should be able to get messages from any channel
uuid = generate_test_order(mock_new_invoice,
client,
order_status=OrderStatus.sent,
tx_seq_num=1,
channel=channel,
admin=True)['uuid']
rv = client.get(f'/admin/message/1?channel={channel}')
assert rv.status_code == HTTPStatus.OK
received_message = rv.data
check_received_message(uuid, received_message)
@patch('orders.new_invoice')
@pytest.mark.parametrize("channel", [
constants.GOSSIP_CHANNEL, constants.BTC_SRC_CHANNEL, constants.AUTH_CHANNEL
])
def test_post_order_unauthorized_channel(mock_new_invoice, client, channel):
# users are not authorized to post to some channels
n_bytes = 500
bid = bidding.get_min_bid(n_bytes)
msg = rnd_string(n_bytes)
mock_new_invoice.return_value = (True,
new_invoice(1, InvoiceStatus.pending,
bid))
rv = client.post('/order',
data={
'bid': bid,
'message': msg.encode(),
'channel': channel
})
assert rv.status_code == HTTPStatus.UNAUTHORIZED
assert_error(rv.get_json(), 'ORDER_CHANNEL_UNAUTHORIZED_OP')
@patch('orders.new_invoice')
@pytest.mark.parametrize("channel", constants.CHANNELS)
def test_post_order_admin(mock_new_invoice, client, channel):
# the admin should be allowed to POST messages to any channel
n_bytes = 500
bid = bidding.get_min_bid(n_bytes)
msg = rnd_string(n_bytes)
mock_new_invoice.return_value = (True,
new_invoice(1, InvoiceStatus.pending,
bid))
rv = client.post('/admin/order',
data={
'bid': bid,
'message': msg.encode(),
'channel': channel
})
assert rv.status_code == HTTPStatus.OK
check_upload(rv.get_json()['uuid'], msg)
@patch('orders.new_invoice')
def test_confirm_tx_missing_or_invalid_param(mock_new_invoice, client):
post_rv = client.post('/order/tx/1')
@ -798,9 +974,9 @@ def test_try_to_pay_a_non_pending_order(mock_new_invoice, client, status):
rv = client.post(f'/callback/{invoice.lid}/{charged_auth_token}')
assert rv.status_code == HTTPStatus.OK
# refetch the order and the invoice from the database
# expecation is that invoice changes its status to paid becasue
# it had the pending status, but order keeps its current status
# Refetch the order and the invoice from the database.
# The expectation is that invoice changes its status to paid because
# it had the pending status, but order keeps its current status.
db_invoice = Invoice.query.filter_by(lid=invoice.lid).first()
db_order = Order.query.filter_by(uuid=uuid_order).first()
assert db_invoice.status == InvoiceStatus.paid.value

43
server/tests/test_orders.py
View File

@ -14,6 +14,7 @@ import constants
import server
from common import new_invoice, place_order, generate_test_order
from error import assert_error
@pytest.fixture
@ -121,7 +122,7 @@ def test_get_orders_limit_parameter(mock_new_invoice, client, state):
state == 'queued' else OrderStatus[state].value
db.session.commit()
# no limit parameter, max PAGE_SIZE should be retunred
# no limit parameter, max PAGE_SIZE should be returned
get_rv = client.get(f'/orders/{state}')
assert get_rv.status_code == HTTPStatus.OK
get_json_resp = get_rv.get_json()
@ -144,6 +145,46 @@ def test_get_orders_limit_parameter(mock_new_invoice, client, state):
assert get_rv.status_code == HTTPStatus.BAD_REQUEST
@patch('orders.new_invoice')
@pytest.mark.parametrize("state", ['pending', 'queued', 'sent'])
@pytest.mark.parametrize("channel", constants.CHANNELS)
def test_get_orders_channel_parameter(mock_new_invoice, client, state,
channel):
n_bytes = 500
mock_new_invoice.return_value = (True,
new_invoice(1, InvoiceStatus.pending,
bidding.get_min_bid(n_bytes)))
# Place all orders as the admin
for i in range(constants.MAX_PAGE_SIZE + 1):
post_rv = place_order(client, n_bytes, channel=channel, admin=True)
assert post_rv.status_code == HTTPStatus.OK
uuid = post_rv.get_json()['uuid']
db_order = Order.query.filter_by(uuid=uuid).first()
db_order.status = OrderStatus.transmitting.value if \
state == 'queued' else OrderStatus[state].value
db.session.commit()
# Get the orders of each channel as a regular user
for _channel in constants.CHANNELS:
get_rv = client.get(f'/orders/{state}?channel={_channel}')
if 'get' in constants.CHANNEL_INFO[_channel].user_permissions:
assert get_rv.status_code == HTTPStatus.OK
get_json_resp = get_rv.get_json()
n_expected_res = constants.PAGE_SIZE if _channel == channel else 0
assert len(get_json_resp) == n_expected_res
else:
assert get_rv.status_code == HTTPStatus.UNAUTHORIZED
assert_error(get_rv.get_json(), 'ORDER_CHANNEL_UNAUTHORIZED_OP')
# Get the orders of each channel as an admin user
for _channel in constants.CHANNELS:
get_rv = client.get(f'/admin/orders/{state}?channel={_channel}')
assert get_rv.status_code == HTTPStatus.OK
get_json_resp = get_rv.get_json()
n_expected_res = constants.PAGE_SIZE if _channel == channel else 0
assert len(get_json_resp) == n_expected_res
@patch('orders.new_invoice')
def test_get_pending_orders(mock_new_invoice, client):
# make some orders

165
server/tests/test_transmitter.py
View File

@ -4,7 +4,7 @@ from datetime import datetime, timedelta
from unittest.mock import patch
from common import generate_test_order, pay_invoice, confirm_tx
from constants import InvoiceStatus, OrderStatus
from constants import InvoiceStatus, OrderStatus, USER_CHANNEL
import transmitter
from database import db
from schemas import order_schema
@ -150,7 +150,9 @@ def test_startup_sequence(mock_new_invoice, client, mockredis):
mock_new_invoice,
client,
invoice_status=InvoiceStatus.paid,
order_status=OrderStatus.transmitting)['uuid']
order_status=OrderStatus.transmitting,
started_transmission_at=datetime.utcnow() -
timedelta(minutes=5))['uuid']
# create two paid orders
first_sendable_order_uuid = generate_test_order(mock_new_invoice,
@ -275,7 +277,7 @@ def test_retransmission(mock_new_invoice, client, mockredis):
# Check the next order for retransmission, which should be the highest
# bidder among the two with pending retransmission (the second order)
order, retry_info = get_next_retransmission()
order, retry_info = get_next_retransmission(USER_CHANNEL)
assert order and retry_info
assert order.id == second_order.id
assert retry_info.order_id == second_order.id
@ -360,3 +362,160 @@ def test_retransmission(mock_new_invoice, client, mockredis):
assert len(retry_order) == 1
assert retry_order[0].order_id == first_order.id
assert retry_order[0].retry_count == 3
@patch('orders.new_invoice')
def test_multichannel_transmission(mock_new_invoice, client, mockredis):
# Post orders on the gossip and btc-src channels via the admin endpoint
gossip_order_uuid1 = generate_test_order(mock_new_invoice,
client,
order_id=1,
bid=1000,
channel=constants.GOSSIP_CHANNEL,
admin=True)['uuid']
gossip_order_uuid2 = generate_test_order(mock_new_invoice,
client,
order_id=2,
bid=1000,
channel=constants.GOSSIP_CHANNEL,
admin=True)['uuid']
btc_order_uuid1 = generate_test_order(mock_new_invoice,
client,
order_id=3,
bid=2000,
channel=constants.BTC_SRC_CHANNEL,
admin=True)['uuid']
btc_order_uuid2 = generate_test_order(mock_new_invoice,
client,
order_id=4,
bid=2000,
channel=constants.BTC_SRC_CHANNEL,
admin=True)['uuid']
# Post regular user-channel orders (requiring payment)
user_order_uuid1 = generate_test_order(mock_new_invoice,
client,
order_id=5,
bid=1000)['uuid']
user_order_uuid2 = generate_test_order(mock_new_invoice,
client,
order_id=6,
bid=2000)['uuid']
gossip_db_order1 = \
Order.query.filter_by(uuid=gossip_order_uuid1).first()
btc_db_order1 = \
Order.query.filter_by(uuid=btc_order_uuid1).first()
user_db_order1 = \
Order.query.filter_by(uuid=user_order_uuid1).first()
# The first admin orders should immediately move to the transmitting state.
# The second admin orders are in paid state (auto/forcedly paid) and
# waiting. The user orders are both in pending state until payment.
assert_order_state(gossip_order_uuid1, 'transmitting')
assert_order_state(gossip_order_uuid2, 'paid')
assert_order_state(btc_order_uuid1, 'transmitting')
assert_order_state(btc_order_uuid2, 'paid')
assert_order_state(user_order_uuid1, 'pending')
assert_order_state(user_order_uuid2, 'pending')
# Pay for the first user-channel order and ensure it moves to the
# transmitting state while the ongoing transmissions in other channels
# remain in progress simultaneously
pay_invoice(user_db_order1.invoices[0], client)
assert_order_state(user_order_uuid1, 'transmitting')
assert_order_state(gossip_order_uuid1, 'transmitting')
assert_order_state(btc_order_uuid1, 'transmitting')
# Calling tx_start should have no impact on the state
transmitter.tx_start()
transmitter.tx_start(constants.GOSSIP_CHANNEL)
transmitter.tx_start(constants.BTC_SRC_CHANNEL)
assert_order_state(gossip_order_uuid1, 'transmitting')
assert_order_state(gossip_order_uuid2, 'paid')
assert_order_state(btc_order_uuid1, 'transmitting')
assert_order_state(btc_order_uuid2, 'paid')
assert_order_state(user_order_uuid1, 'transmitting')
assert_order_state(user_order_uuid2, 'pending')
# Next, assume the Tx hosts send Tx confirmations
confirm_tx(gossip_db_order1.tx_seq_num, all_region_numbers, client)
confirm_tx(btc_db_order1.tx_seq_num, all_region_numbers, client)
confirm_tx(user_db_order1.tx_seq_num, all_region_numbers, client)
# Once confirmed, the next admin orders should start automatically. The
# next user order does not start because the payment is still missing.
assert_order_state(gossip_order_uuid1, 'sent')
assert_order_state(gossip_order_uuid2, 'transmitting')
assert_order_state(btc_order_uuid1, 'sent')
assert_order_state(btc_order_uuid2, 'transmitting')
assert_order_state(user_order_uuid1, 'sent')
assert_order_state(user_order_uuid2, 'pending')
def generate_paid_test_orders():
user_channel_order_1 = Order(uuid='uuid_user',
unpaid_bid=2000,
message_size=10,
message_digest='some digest',
status=OrderStatus.paid.value)
gossip_order = Order(uuid='uuid_gossip',
unpaid_bid=2000,
message_size=10,
message_digest='some digest',
status=OrderStatus.paid.value,
channel=constants.GOSSIP_CHANNEL)
btc_order = Order(uuid='uuid_btc',
unpaid_bid=2000,
message_size=10,
message_digest='some digest',
status=OrderStatus.paid.value,
channel=constants.BTC_SRC_CHANNEL)
auth_order = Order(uuid='uuid_auth',
unpaid_bid=2000,
message_size=10,
message_digest='some digest',
status=OrderStatus.paid.value,
channel=constants.AUTH_CHANNEL)
db.session.add(user_channel_order_1)
db.session.add(gossip_order)
db.session.add(btc_order)
db.session.add(auth_order)
db.session.commit()
def test_tx_start_with_single_channel(client):
generate_paid_test_orders()
transmitter.tx_start(constants.USER_CHANNEL)
assert_order_state('uuid_user', 'transmitting')
assert_order_state('uuid_gossip', 'paid')
assert_order_state('uuid_btc', 'paid')
assert_order_state('uuid_auth', 'paid')
transmitter.tx_start(constants.GOSSIP_CHANNEL)
assert_order_state('uuid_user', 'transmitting')
assert_order_state('uuid_gossip', 'transmitting')
assert_order_state('uuid_btc', 'paid')
assert_order_state('uuid_auth', 'paid')
transmitter.tx_start(constants.BTC_SRC_CHANNEL)
assert_order_state('uuid_user', 'transmitting')
assert_order_state('uuid_gossip', 'transmitting')
assert_order_state('uuid_btc', 'transmitting')
assert_order_state('uuid_auth', 'paid')
transmitter.tx_start(constants.AUTH_CHANNEL)
assert_order_state('uuid_user', 'transmitting')
assert_order_state('uuid_gossip', 'transmitting')
assert_order_state('uuid_btc', 'transmitting')
assert_order_state('uuid_auth', 'transmitting')
def test_tx_start_without_channel(client):
generate_paid_test_orders()
transmitter.tx_start()
assert_order_state('uuid_user', 'transmitting')
assert_order_state('uuid_gossip', 'transmitting')
assert_order_state('uuid_btc', 'transmitting')
assert_order_state('uuid_auth', 'transmitting')

53
server/transmitter.py
View File

@ -3,6 +3,7 @@ import logging
from datetime import datetime
from flask import current_app
from sqlalchemy import and_
import constants
import order_helpers
@ -35,27 +36,51 @@ def publish_to_sse_server(order, retransmit_info=None):
msg['regions'] = region_code_to_number_list(
retransmit_info.region_code)
msg = json.dumps(msg)
redis().publish(channel=constants.SUB_CHANNELS[0], message=msg)
redis().publish(channel=constants.CHANNEL_INFO[order.channel].name,
message=msg)
return
def tx_start():
transmitting_orders = Order.query.filter_by(
status=constants.OrderStatus.transmitting.value).all()
def tx_start(channel=None):
"""Look for pending transmissions and serve them
An order is ready for transmission when already paid or when being
retransmitted. Also, a pending transmission can only be served if there is
no other ongoing transmission on the logical channel. Each channel can only
serve one transmission at a time.
This function works both for a single defined channel or for all channels.
When the channel parameter is undefined, it looks for pending transmissions
in all channels. Otherwise, it processes the specified channel only.
Args:
channel (int, optional): Logical transmission channel to serve.
Defaults to None.
"""
# Call itself recursively if the channel is not defined
if (channel is None):
for channel in constants.CHANNELS:
tx_start(channel)
return
transmitting_orders = Order.query.filter(
and_(Order.status == constants.OrderStatus.transmitting.value,
Order.channel == channel)).all()
# Do not start a new transmission if another order is being transmitted
# right now
if len(transmitting_orders) > 0:
return False
return
# First, try to find a paid order with the highest bid in the orders table
# and start its transmission. If no orders are found there, look into the
# tx_retries table and retransmit one of the orders from there if it meets
# the retransmission criteria
order = Order.query.filter_by(
status=constants.OrderStatus.paid.value).order_by(
Order.bid_per_byte.desc()).first()
# and start its transmission. If no paid orders are found there, look into
# the tx_retries table and retransmit one of the orders from there if it
# meets the retransmission criteria
order = Order.query.filter(
and_(Order.status == constants.OrderStatus.paid.value,
Order.channel == channel)).order_by(
Order.bid_per_byte.desc()).first()
if order:
logging.info(f'transmission start {order.uuid}')
@ -67,7 +92,7 @@ def tx_start():
else:
# No order found for the first transmission.
# Check if any order requires retransmission.
order, retransmit_info = order_helpers.get_next_retransmission()
order, retransmit_info = order_helpers.get_next_retransmission(channel)
if order and retransmit_info:
logging.info(f'retransmission start {order.uuid}')
order.status = constants.OrderStatus.transmitting.value
@ -89,4 +114,4 @@ def tx_end(order):
db.session.commit()
publish_to_sse_server(order, retransmit_info)
# Start the next queued order as soon as the current order finishes
tx_start()
tx_start(order.channel)

2
terraform/modules/blc/cloud-init/blc.yaml
View File

@ -265,7 +265,7 @@ write_files:
--network=host \
--pid=host \
--name=sse-server \
-e "SUB_CHANNELS=transmissions" \
-e "SUB_CHANNELS=transmissions,gossip,btc-src,auth" \
-e "REDIS_URI=redis://localhost:6379" \
"${ionosphere_sse_docker}"
ExecStop=/usr/bin/docker stop sse-server

9
terraform/modules/lb/cloud-init/lb.yaml
View File

@ -40,7 +40,7 @@ write_files:
location / {
rewrite ^ https://$http_host$request_uri? permanent;
}
location ~ ^/auth/order($|/.*)$ {
# Restrict auth/order server endpoints to local subnets only
allow 127.0.0.1;
@ -119,6 +119,13 @@ write_files:
proxy_pass http://${mainnet_ip}:9292/order/rx/;
}
location /admin/ {
if ($ssl_client_verify != SUCCESS) {
return 403;
}
proxy_pass http://${mainnet_ip}:9292/admin/;
}
# Proxy to mainnet Satellite API
location / {
add_header 'Access-Control-Allow-Origin' '*' always;