Mélodie Damien Simonin Feugas

Melodie is a portable, simple-as-pie music player

README

Mélodie

Melodie is a portable, simple-as-pie music player.

There are thunsands of them in the wild. This mine is an excuse for learning Electron, Svelte and reactive programming.

Installation

You will find other installers on the releases page.

Please note that AppImage Snap and NSIS installer will automatically update to the latest available version.

If you run Mélodie from a zip or using DMG/Windows portable version, you will have to download updates by yourself.

Note for Windows users

Windows installers are not signed.

When you will run the .exe files, Windows will warn you that the source is insecure (it is not!).

It is possible to bypass the warning by clicking on the "More information" link, then on the Install button

If you install the app through the Windows App Store, you'll get no warning, since the store team reviewed and approved it.

Note for MacOS users

DMG image is not signed.

After you will have downloaded the .dmg file, open it and drag the Mélodie icon to the Application Icon. Then, MacOS will prevent you from opening Mélodie as I haven't paid for an app deployment certificate.

Once you will have closed the annoying warning, open you Security panel in settings, and go to General tab. There, you should see the list of recently blocked application: Mélodie should be there.

You can add it as an exception, and then run it (see: How to open an app that hasn’t been notarized or is from an unidentified developer).

Another option is to open it with Control-click: it'll immediately register the app as an exception (see: Open a Mac app from an unidentified developer).

TODOs

features

• use file and folder names to complete missing tags

• merge components/Album|Artist|Playlist tests for GridItem + hover behaviour (desktop only)

• play all button

• indicates when track is in playlist

• configure replay gain from settings

• display tracks/albums/artists count in settings

• allow reseting database from settings

• list images from track tags when collecting candidate covers for an album

• progressive webapp

• Consider yarn2/pnpm, once svelte-preprocess is fixed

• search tooling to find deps version mismatch, and maintain package.json same version

• compare ajv serialization with stringify

• accessibility: ImageUploader file input, Loading input, and Nav search box have no label

• download files and cache them in browser

tools

• dependabot + dep update

• automated end-to-end tests

• more technical documentation (install & release process notably)

Bugs and known issues

1. When server is not reachable, attempts to establish new WebSocket connection takes longer and longer

2. DMG package does not download updates: it requires zip, and we cannot build zip because of the accent in product name...

3. Playlist models are not updated on tracks removal

4. Undetected live changes: remove tracks and re-add them. This is a linux-only issue with chokidar

   • Issue #917

   • Issue #591

5. When loading new folders, enqueuing or going to album details will give incomplete results. Going back and forth won't load new data

6. Security: clean html in artist/album names (wrapWithRefs returns injectable markup)

7. AppImage, when used with AppImageLauncher, fail to auto update

8. If we knew current position in browser history, then we could disabled navigation button accordingly

9. Page navigation: use:link doesn't work in tests and raise Svelte warning. a.href is fine

10. Disklist/TrackTable dropdown does not consider scroll position (in storybook only)

11. Testing input: fireEvent.change, input or keyUp does not trigger svelte's bind:value on input

12. The test suite is becoming brittle

    1. Media service › triggerAlbumsEnrichment › saves first returned cover for album

       > 839 |       expect(await fs.readFile(savedAlbums[0].media, 'utf8')).toEqual(
             |                                                               ^
       

    2. Media service › triggerAlbumsEnrichment › retries album with no cover but at least one restriced provided Is a 1ms difference in expected processedEpoch

    3. AddToPlaylist component › given some playlists › saves new playlist with all tracks The dropdown menu is still visible (probably because of the animation)

    4. snackbars store > showSnack > uses the specified duration when enqueuing slacks

       - Expected  - 3
       + Received  + 0
       
       > 96 |       expect(snackbarCalls).toEqual([
            |                             ^
       

13. The Media test do not pass on Windows: nock is not giving recorded bodies

Data

Mélodie is using SQLite3 to store settings, playlists and tracks's metadatas. SQLite3 stores everything in a single file, named db.sqlite3 and located into the application userData folder.

Mélodie also stores artists artwork according to the ARTWORK_DESTINATION environment variable, sets to user's pictures folder in melodie-media folder.

Configuring logs

Log are written to a file, which location is set by LOG_DESTINATION env variable. Mélodie Desktop sets LOG_DESTINATION to logs.txt in the application logs path.

Log levels are configured in a file defined by LOG_LEVEL_FILE env variable. Mélodie Desktop sets it to .levels in the application userData folder.

Its syntax is:

# this is a comment
logger-name=level
wildcard*=level

logger names are:

• core

• renderer

• updater

• services/ where is tracks, playlists, media, settings

• providers/ where is local, audiodb, discogs

• models/ where is tracks, albums, artists, playlists, settings

and levels are (in order): trace (most verbose), debug, info, warn, error, fatal, silent (no logs)

Wildcards can be at the beginning *tracks or the end models/*. In case a logger name is matching several directives, the first always wins.

You can edit the file, and trigger logger level refresh by sending SIGUSR2 to the application: kill -USR2 {pid} (first log issued contains pid)

Running locally

You'll need npm@7+ and node@16+

git clone git@github.com:feugy/melodie.git
cd melodie
npm i
npm start

Testing

The test suite works fine Linux, MacOS and Windows.

npm t

Core services network mocks (nocks)

Some services are hitting external APIs, such as AudioDB. As we don't want to flood them with test requests, these are using network mocks.

To use real services, run your tests with REAL_NETWORK environment variables (whatever its value). When using real services, update the mocks by defining UPDATE_NOCKS environment variables (whatever its value). Nocks will stay unchanged on test failure.

Some providers need access keys during tests. Just make a .env file in the root folder, with the appropriate values:

DISCOGS_TOKEN=XYZ
AUDIODB_KEY=1

Promo site

Located under apps/site, it publicize Mélodie and has a button to download latest release artifacts.

Caveats:

• it is build for Github pages. That is, the final hosting will have a base path (/melodie), which we don't have when trying out locally
• it reuses common/ui stylesheet, which requires workaround due to font paths. Fonts must be copied into /apps/site/static/fonts for dev, and the production build creates undesired copies in /common/ui/src/fonts
• when developing locally, use npm -w apps/site run dev and browse to http://localhost:3000
• when willing to try out production result, use npm -w apps/site run build, npm -w apps/site run serve and browse to http://localhost:3000/melodie

Trying snaps out

Working with snaps locally isn't really easy.

1. install the real app from the store:

   snap install melodie
   

2. then package your app in debug mode, to access the unpacked snap:

   DEBUG=electron-builder npm run release:artifacts --workspace apps/desktop -- -l
   

3. copy missing files to the unpacked snap, and keep your latest changes:

   mkdir dist/__snap-amd64/tmp
   mv dist/__snap-amd64/* dist/__snap-amd64/tmp
   cp -r /snap/melodie/current/* dist/__snap-amd64/
   cp -r dist/linux-unpacked/* dist/__snap-amd64/
   mv dist/__snap-amd64/tmp/* dist/__snap-amd64/*
   

4. now use your development code:

   snap try dist/__snap-amd64
   melodie
   

5. and revert when you're done:

   snap revert melodie
   

Checking AppImage

To check that generated AppImage works:

1. Install AppImageLauncher if not done yet

2. Download AppImageLint

3. Package application for linux

   npm run release:artifacts --workspace apps/desktop -- -l
   

4. Lint your AppImage:

   appimagelint dist/Mélodie.AppImage
   

5. Double click on ./dist/Mélodie.AppImage and integrate it to your system. Please check that the app starts, it can access to local files, its name and icon are correct in the launcher

Releasing

Release process is fairly automated: it will generate changelog, bump version, and build melodie for different platform, creating several artifacts which are either packages (snap, AppImage, Nsis, appx) or plain files (zip).

Theses artifacts will be either published on their respective store (snapcraft, Windows App store...) or uploaded to github as a release. Once a Github release is published, users who installed an auto-updatable package (snap, AppImage, Nsis, appx) will get the new version auto-magically.

Windows App store release can not be automated: Github CI will build the appx package, but it must be manually submitted to the Windows App store.

1. When ready, bump the version on local machine:

   npm run release:bump
   

   (if you wish to get a pre-release, append -- --prerelease beta|alpha to the command line)

2. Don't forget to update snapshots: the presentation site test depend on the version number.

   TAG=$(git describe --tags)
   TAG=${TAG::-11}
   npm t --workspace apps/site -- --clearCache
   npm t --workspace apps/site -- -u
   git commit -a --amend --no-edit
   git tag -f $TAG
   

   You shoud see 2 snapshots updated

3. Then push tags to github, as it'll trigger the artifact creation:

   git push --follow-tags
   

4. Finally, go to github releases, and edit the newest one:

   1. give it a code name

   2. copy the latest section of the changelog in the release body

   3. save it as draft

   4. Wait until the artifacts are published on your draft

   5. manually submit the new appx package to the Windows App store

   6. remove the appx package from artifact list: as it is unsigned, users can not install it from here

   7. publish your release

   8. go and slack off!

Manual snap release

Until this issue on Github Actions is fixed on Electron-builder, we're stuck with electron-builder@22.10.5 and we need to manually release on snap.

1. Clean up distribution, build snap file and extract it:

   rm -rf apps/desktop/dist/
   npm -w apps/desktop  run release:artifacts -- -l snap
   cd apps/desktop/dist/
   rm -rf linux-unpacked builder-effective-config.yaml
   file-roller -f *.snap .
   

   Then select the dist folder as target folder, and close the "Could not open 'dist'" error popup.

2. Amend the meta/snap.yaml descriptor. At root level, replaces slots with:

   slots:
     mpris:
       interface: mpris
       name: chromium
   

   Then within app.melodie.slots, remove - name

   Save the file

3. Re-create snap file and publish it on snapcraft:

   rm -r *.snap
   snapcraft pack . --output 'linux - Mélodie.snap'
   snapcraft login
   snapcraft upload --release=stable 'linux - Mélodie.snap'
   

Publicise

Mélodie is referenced on these stores and hubs:

• Electron's app list (PR)

• Svelte's showcase (PR)

• Snap store (app page)

• AppImage hub (PR)

• Windows App store (app page)

Notable facts

• Started with a search engine (FlexSearch) to store tracks, and serialized JS lists for albums & artists. Altough very performant (50s to index the whole music library), the memory footprint is heavy (700Mo) since FlexSearch is loading entire indices in memory

• Moved to sqlite3 denormalized tables (drawback: no streaming supported)

• Dropped the idea to query tracks of a given albums/artists/genre/playlist by using SQL queries. Sqlite has a very poor json support, compared to Postgres. There is only one way to query json field: json_extract. It is possible to create indexes on expressions, and this makes retrieving tracks of a given album very efficient:

  create index track_album on tracks (trim(lower(json_extract(tags, '$.album'))))
  select id, tags from tracks where trim(lower(json_extract(tags, '$.album'))) = lower('Le grand bleu')
  

  However, it doesn't work on artists or genres, because they are modeled with arrays, and operator used do not leverage any index:

  select id, tags from tracks where instr(lower(json_extract(tags, '$.artists')), 'eric serra')
  select id, tags from tracks where json_extract(tags, '$.artists') like '%eric serra%'
  

• chokidar is the best of breed watch tool, but has this annoying linux-only big when moving folders outside of the watched paths Watchman is a C program that'll be hard to bundle. node-watch does not send file event when removing/renaming folders watchr API seems overly complex watch-pack is using chokidar and the next version isn't ready

• wiring jest, storybook, svelte and tailwind was really painfull. Too many configuration files now :( To make storyshots working, I had to downgrade Jest because of an annoying bug (reference).

• I considered Sapper for its nice conventional router, but given all the unsued feature (service workers, SSR) I chose a simpler router. It is based on hash handling, as electron urls are using file:// protocol which makes it difficult to use with history-based routers.

• Initially, albums & artists id where hash of their names. It was very convenient to keep a list of artist's albums just by storing album names in artist's linked array. UI would infer ids by applying the same hash. However, it is common to see albums with same name from different artists (like "Greatest hits"). To mitigate this issue, I had to make album's id out of album name and album artist (when defined). This ruined the hash convention, and I had to replace all "links" by proper references (id + name). Now UI does not infer ids anymore.

• For system notifications, document.hidden and visibilityChange are too weak because they only notice when the app is minimized/restored

• System notification was tricky: HTML5 Notification API doesn't support actions, except from service workers. Using service workers was overkill, and didn't work in the end. Electron's native notificaiton does not support actions either. Using node-notifier was a viable possibility, but doesn't support actions in a portable fashion (notify-send on linux doesn't support it). Finally back to HTML5 notification API, without actions :(

• The discovery of mediaSession's metadata and handler was completely random. It's only supported by Chrome (hopefully for me!), and can be seen on Deezer, Spotify or Youtube Music. However, it does not display artworks.

• IntersectionObserver does not call the intersection entry when the position inside viewport is changing but the intersection doesn't. As a result, dropdown in the sheet will enter viewport during sheet animation, causing troubles positioning the menu

• AC/DC was displayed as 2 different artists ('AC' and 'DC'). This is an issue with ID3 tags: version 2.3 uses / as a separators for artists. Overitting mp3 tags with 2.4 solved the issue

• Snap packaging was hairy to figure out. It is clearly the best option on Linux, as it has great desktop integration (which AppImage lacks) and a renowed app store. However, getting the MediaMetadata to work with snap confinement took two days of try-and-fail research. The full journey is available in this PR on electron-builnder. Besides, the way snapd is creating different folders for each new version forced me to move artist albums outside of electron's data folders: snapd ensure that files are copied from old to new version, but can not update the media full paths store inside SQLite DB.

• MacOS builder was constantly failing with the same error: 7zip couldn't find any file to compress in the final archive. It turns out it is because the production name as an accent (Mélodie), and the mac flavor of 7zip can not handle it...

• Chokidar has a "limitation" and triggers for each renamed or moved file an 'unlink' and an 'add' event. The implication on Mélodie were high: moved/renamed files would disappear from playlists. Ty bypass the issue, Mélodie stores file inodes and buffer chokidar events: when a file is removed, Mélodie will wait 250ms more, and if another file is added with the same inode during that time, will consider it as a rename/move.

• The mono-repo endeavour. My goal was to split code in various reusable packages: a UI and core that would not depend on Electron, and could be used in both Web and Desktop context, and two apps: an Electron-based desktop application and the Github-page site. As developer I would expect the ability to hoist as many modules

  • runing jest with pnpm does not work at all.
  • lerna is a pain when it comes to hoisting deps.
  • svelte-jester and preprocess absolutely don't work with yarn@2
  • yarn@1 works fine but brings very little commands (just a little more than npm@7)
  • npm@7 < 7.24 must install peer deps in legacy mode and does not offer any sugar for multi-package commands. All deps must be manually added to package.json, because install command MUST be run at root level Electron-builder does not like monorepo either: author, description and other metadata must be copied from root package.json to apps/desktop/package.json. The Electron version must be fixed because node_modules are hoisted. The package.json name MUST be melodie :( Caveats: always run npm i --legacy-peer-deps AT ROOT level. Running npm or npx command inside packages would re-create node_modules Ensuring the same version in all packages and dependencies similarities must be done manually

• svelte-spa-router, and its dependency on regexparam, has been bother me for a very long time. When ran with jest, svelte-spa-router files must be transpiled by Svelte compiler, but they import regexparam as esm, and this lib doesn't expose such binding. One must replace the import with require, and this must only be done during test, because rollup will handle it properly. When receiving errors from svelte-jester, don't forget to clean jest cache with --cleanCache CLI option.

• since v22.11.1, electron-builder fails to build the app on Github worker. Fixing the version to 22.10.5 for the time being.

• Tailwind is veeeeeeeeeeeery slow to compile. Svelte preprocessor can not handle it fast, making vite pretty slow when starting atelier (only the first load). More information here. Moving to Windi CSS speed the build time from 65 to 28 seconds!

• The Audio element failed to play any music when coupled with AudioContext:

  1. Bluetooth must be enabled prior to starting the app (simply reload the app once enabled)
  2. AudioContext will build, but will not process any data. Being running or suspended (as per Google's policy) does not matter: rebuilding the context or building it on user interaction does not solve the issue as long as bluetooth is enabled

How watch & diff works

• on app load, trigger diff

  1. get followed folders from store

  2. crawl followed folders, return array of paths + hashs + last changed

  3. get array of tracks with hash + last changed from DB

  4. compare to find new & changed hashes

     1. enrich with tags & media

     2. save

  5. compare to isolate deleted hashes

     1. remove corresponding tracks

• while app is running

  1. watch new & changed paths

     1. compute hash, enrich with tags & media

     2. save

  2. watch deleted paths

     1. compute hash

     2. remove corresponding tracks

• when adding new followed folder

  1. save in store

  2. crawl new folder, return array of paths

  3. compute hash, enrich with tags & media

  4. save

How missing artworks/covers retrieval works

• on UI demand trigger process

  1. push all artists/albums without artwork/cover, and not process since N in a queue

  2. apply rate limit (to avoid flooding disks/providers)

  3. call providers one by one

     1. save first result as artwork/cover, stop

     2. on no results, but at least on provider returned rate limitation, enqueue artist/album

     3. on no results, save date on artist/album