Skip to content

Sevelantis/distributed-computing-audiosystem

Repository files navigation

Goal

University of Aveiro

The goal of this project was to implement a distributed system to process time-costly audio processing jobs, by distributing them across many remote workers.

The end-user doesn't need to wait years, but seconds.

Prominent project highlights

  • Fanout exchange queue - allows to send exactly one message to every subscribed client
  • Fault Tolerance - ACK/NACK messages, adjusting clients' heartbeats
  • Performance - Identifying bottlenecks is the first thing to consider to tune performance.

Dependencies

For Ubuntu (and other debian based linux), run the following commands:

sudo apt install ffmpeg
sudo apt install make

setup local python environment

Run the following commands to setup the environement:

python3 -m venv venv
source venv/bin/activate

pip install pip --upgrade
python3 -m venv --upgrade venv
pip install -r requirements.txt

setup the broker

  • docker-compose up -d
  • IMPORTANT!!! - Create an exchange queue using the broker's UI!

Start the server

export PYTHONPATH=$PWD
make server

Start workers

  • You can start the workers to run on a different machine.
  • Before run, since they subscribe to the broker, make sure to set the broker IP in the src/config.py file. E.g. BROKER_HOST = '192.168.0.4'. By default it is localhost, if the server and the workers run on the same machine.
export PYTHONPATH=$PWD
make workers

Usage

  1. Open the Web Page. Open templates/index.html file using browser.
  2. Drop an MP3 file, and wait for the progress to go 100% (don't refresh the page while processing, the output will disappear), afterwards you can download your files:
    • Separated drums track
    • Separated vocals track
    • Separated other instruments (mostly guitar + keyboard) track
    • Separated bass track
    • Mix of chosen instruments (default is drums + bass)

This project is licensed under the MIT License - see the LICENSE file for details.

Fault Tolerance

  • This system is resillient to worker crashes, please see the WORKERS_UNHEALTHY parameter to simulate the crash after WORKER_SIMULATE_CRASH_AFTER time.

Configurations

  • in src/config.py you can find a number of parameters to be adjusted, e.g.:
### User Conf ###
BROKER_HOST = 'localhost'           # broker IP
WORKERS_HEALTHY = 10                # Amount of workers that do not crash
WORKERS_UNHEALTHY = 2               # Amount of workers that crash (their process terminates)
WORKER_SIMULATE_CRASH_AFTER = 20    # After this amount of seconds.
### User Conf ###

Techstack

  • Backend: FastAPI
  • Middlewares: RabbitMQ
  • Frontend: Bootstrap, html, css

Authors

Student: Miron Oskroba

NMEC: 112169

University of Aveiro

Computação Distribuída