The Consolidated API: Progress and Updates

Introduction

Since the advent of our goal to simplify Resonate’s tech stack and creating a functional development environment that’s easy for newcomers to get up and running (so they can contribute!), and in line with Product Development Update: October 2022, we’ve been hard at work on the consolidated API. (I realized that we have specific forum threads for front-end clients such as Beam: A Native App For Linux, macOS, and Windows (testing please!) and Minimal mobile apps for Android and iOS to track progress and report back to the community how things are going, we didn’t yet have a thread going for the the back-end API, so the aim of this thread is to fulfill that role.)

What is an API?

What is a API or a back-end in non-tech-speak terminology? Think of it kind of like how a public library works perhaps: if you go to the library looking for a particular book, you can ask a librarian to help you find, and check out something that you’re looking for. They swipe your library card (authentication / logging in), and you can take the book with you, or make copies of pages of a book to keep. Think of the API as the librarian, serving requests for specific pieces of data (music to stream, album art to look at, music metadata, etc.).

Consolidation

So instead of splitting up these services across many different and separate folders with code you have to download separately and set up separately to get running as a developer, the goal here is for one download and everything gets up and running with a simple command - we want developers to be able to focus on contributions and improving our codebase, rather than worrying about how to get everything working.

Progress as of Mid-October

Work has been moving quickly on the API (tracked in our Product Backlog), and in recent weeks we’ve introduced extensive testing, which lets developers know immediately if the changes they’re attemping to make will break things (essential for newcoming devs looking to contribute but don’t yet understand the codebase in full).

I want to give a massive shout out to @psi for putting forth an immense amount of dev-ops work and elbow-grease into this, as well as @matt_burnett for putting in a lot of incredible work with the testing: which with the above additions means that we should now have tests running every time someone puts in a change request to our code. It’s not a full test suite yet, but this will give us a lot of insight into how and when we change things, and how they affect the overall API. They’ll catch unexpected side-effects across the app, and means you don’t need to have one person holding the knowledge of the entire app to prevent side-effect you didn’t expect.

The tests will also make it much easier for us to know what output we’re expecting from endpoints, and to write endpoints so that they conform to that output. Whenever we make changes to the API, or fixing bugs, we should be writing tests for those changes. (should is an ideal world, realize we are a volunteer project).

This may be a bit wobbly, as @psi has noticed that sometimes the GitHub Action doesn’t load in the expected test libraries, for example, so we’ll have to iron that out, but it’s a huge step forward for confidence in that our code is doing what we think it should do (or aspire it to do).

We’ll be sure to continue to post further updates below this initial post, so we can continue to make this work visible/transparent, and receive feedback from the community.

7 Likes

Love this metaphor. Here for the library socialism.

4 Likes

It’s been a second since we’ve updated this thread, so I think it’s a good time to circle back and let everyone know how things are progressing. There’s been a lot of impressive movement on this front, with a lot of people pitching in.

Consolidating the Database

We’ve been consolidating the old MySQL and postgres databases into one. This has been a long and somewhat tedious process, since it’s been a matter of finding what means what, how certain things were stored in the past, and understanding Wordpress’ and its plugins sometimes convoluted database decisions. For progress on this, please check this GitHub issue. Completing this will make a lot of our work much simpler, and set us up for easier queries, better reporting, and better integration with other systems.

For the rest, here’s some high level checklists for the tasks that are outstanding before we can switch completely to this new system.

Production Environment

  • :white_check_mark: Get the production environment running on the same server as where all of our data is stored.
  • :white_check_mark: Images and audio being read from the right location on the server
  • :white_check_mark: Images and audio being uploaded to the right location on the server. This works locally but hasn’t been tested on production.
  • Writing a migration script that moves things from the old databases and servers into the new ones. This is done but it has to be run right before we move.
  • :white_check_mark: Set up an emailing service to send out registration and sign up emails
  • :white_check_mark: Serve the API over a dedicated url
  • :white_check_mark: SSL Encryption functional
  • Adapting Beam to use the new API and databases. This is 95% done but there continue to be edgecases we want to iron out.

General Users & Admin

  • :white_check_mark: On user registration, emails are sent
  • System admin dashboard to allow us to change status of music (private or public, free or paid, etc) and user accounts. This is partially implemented but needs to go through some user testing
  • :white_check_mark: User login functional
  • :white_check_mark: Emails are lowercased on registration to prevent bugs down the line where casing doesn’t match
  • :white_check_mark: Password reset functional
  • :white_check_mark: Purchasing credits via Stripe functional
  • :white_check_mark: Artist self-uploading of tracks and trackgroups

Test Coverage

We have over 100 tests running on our API that run whenever we add new code to ensure that our code changes do not break existing functionality. Big kudos to @matt_burnett for the help with this!

Next

We’re getting close to needing volunteers to help us test the new system and make sure that we’re reaching functionality parity! If that’s you, please DM either @psi or @piper to help out, or post below!

7 Likes

Beyond legendary

3 Likes