Smart Playlists

[deluan]

Smart playlists can be created as a json object into a file with .nsp extension and Navidrome will create a dynamic/query based playlist and keep it updated. These .nsp files will be imported the same way as normal playlists, i.e. at scan time. See comment below for a complete list of fields and operators available.

Some examples:

• Recently Played:

{
  "name": "Recently Played",
  "comment": "Recently played tracks",
  "all": [
    {"inTheLast": {"lastPlayed": 30}}
  ],
  "sort": "lastPlayed",
  "order": "desc",
  "limit": 100
}

• 80's Top Songs

{
  "all": [
    { "any": [
      {"is": {"loved": true}},
      {"gt": {"rating": 3}}
    ]},
    {"inTheRange": {"year": [1981, 1990]}}
  ],
  "sort": "year",
  "order": "desc",
  "limit": 25
}

• Favourites

{
  "all": [
    {"is": {"loved": true}}
  ],
  "sort": "dateLoved",
  "order": "desc",
  "limit": 500
}

• All songs in random order (not recommended for large libraries)

{
  "all": [
    {"gt": {"playCount":-1}}
  ],
  "sort": "random"
}

TODO

• PHASE 1 - On demand

  • Add rules to playlists table
  • Rule compiler (to SQL)
  • Calculate tracks on get
    • Cache results (for 5 secs)
  • Import smart playlists found in Music Folder (extension .NSP)
  • Handle permissions
    • Smart Playlists (tracks) are read-only
    • Cannot be shared
  • Refactor and clean up

• PHASE 2 - Live update

  • Update fields table on smart playlist create/update
  • Live update queue: Persistent queue (SQLite3?) to store id and rules of playlist to process
  • SP queue processor: get id and rules from queue and refresh tracks based on rules
  • UpdateNotifier (in memory)
    • Group updates (10 items or 3 secs idle)
    • Add playlists to live update queue
    • Call it from Media Annotations and Scanner (when new tracks are imported)

• PHASE 3 - UI

  • Rule editor
  • Reorganize sidebar

• MISCELANEOUS

  • Sort by "random"
  • Album and Artist attributes
  • Add a "playlist" field - see how to use it

[deluan]

Current list of fields available (case-insensitive):

title
album
artist
albumartist
hascoverart
tracknumber
discnumber
year
size
compilation
dateadded
datemodified
discsubtitle
comment
lyrics
sorttitle
sortalbum
sortartist
sortalbumartist
albumtype
albumcomment
catalognumber
filepath
filetype
duration
bitrate
bpm
channels
genre
loved
dateloved
lastplayed
playcount
rating

Operators available (case-insensitive):

is
isNot
gt
lt
contains
notContains
startsWith
endsWith
inTheRange
before
after
inTheLast
notInTheLast
inPlaylist
notInPlaylist

[RedKage]

Hello, not sure if I should post this here or make a new issue.
I have a use case which is to find all the songs which are unrated. I want to find those which have no star.

I played a bit with the NSP, and my take is that it's like an expression tree which builds SQL queries. Haven't looked at the source, yet. And indeed, the "problem" is that
select * from annotation where rating = 0 and item_type = 'media_file'
returns only the songs with no rating, but they still need to have a row in the annotation table.

In effect I have only 10 songs returned without star. When in actuality I have thousands, but these have never been annotated.
So they are never in the scope of the expression tree. So they can never be found? Or is there another way?

EDIT: I wonder if there's like some sort of left outer join or something that would still target media_files which have no annotations yet, kinda like changing that LeftJoin in persistence/playlist_repository.go:207

[deluan]

Hello, not sure if I should post this here or make a new issue.

That's fine

I played a bit with the NSP, and my take is that it's like an expression tree which builds SQL queries. Haven't looked at the source, yet. And indeed, the "problem" is that select * from annotation where rating = 0 and item_type = 'media_file' returns only the songs with no rating, but they still need to have a row in the annotation table.

Yes, that's how it works, and yes, this is a "problem". I think a good approach for this is to introduce a new operator, isSet (true/false) that will be converted to is null/is not null and will allow these use cases where you don't have the record in the annotation table.

EDIT: I wonder if there's like some sort of left outer join or something that would still target media_files which have no annotations yet, kinda like changing that LeftJoin in persistence/playlist_repository.go:207

It is already a left join, the issue is that when there's no matching row in the annotation table, its values are all null

[deluan]

I actually did some experimentation and I think a better way is to wrap all fields from the left joined tables with COALESCE, like this:

COALESCE(annotation.rating, 0 ) = 0)

This will allow to use "is": {"rating": 0} in the NSP file.

[0bmay]

First off, thanks for the smart playlists.. It really opens up the possibilities for Navidrome!

using the latest docker image (Navidrome Version: | 0.47.0 (8c7d95c)), trying to use dateadded and datemodified return invalid key errors. I'm attempting to make a Songs newly added and unrated playlist.. I took a cue from the random sorting post on reddit. Not sure if it would work, but it appears I can't filter on dateadded or datemodified. I'm assuming I'm doing something wrong, but just in case....

{
"all": [
{"is": {"rating": 0}},
{"datemodified": {"gt": "DATE('now','-90 day')"}}, -- using this line or the next..
{"dateadded": {"gt": "DATE('now','-90 day')"}}
],
"sort": "dateadded",
"order": "asc",
"limit": 100
}

time="2021-11-20T00:56:54Z" level=error msg="Error parsing SmartPlaylist" error="invalid expression key dateadded" playlist="Songs to Rate - Newly Added" requestId=e31872cd1409/MIUyT3LsdO-001067,
time="2021-11-20T00:56:54Z" level=error msg="Error parsing playlist" error="invalid expression key dateadded" path="/music/playlists/Songs to Rate - Newly Added.nsp" requestId=e31872cd1409/MIUyT3LsdO-001067,
time="2021-11-20T01:02:46Z" level=error msg="Error parsing SmartPlaylist" error="invalid expression key datemodified" playlist="Songs to Rate - Newly Added" requestId=e31872cd1409/MIUyT3LsdO-001108,
time="2021-11-20T01:02:46Z" level=error msg="Error parsing playlist" error="invalid expression key datemodified" path="/music/playlists/Songs to Rate - Newly Added.nsp" requestId=e31872cd1409/MIUyT3LsdO-001108,

[deluan]

Hey @0bmay, the issue is that the key of the rule has to be the operator, not the field. So in the snippet:

{"dateadded": {"gt": "DATE('now','-90 day')"}}

should be:

{"gt": {"dateadded": "DATE('now','-90 day')"}}

Also notice the operator inTheLast, that was created specifically for this type of rules:

{"inTheLast": {"dateadded": "90"}}, 

I don't even know if DATE('now','-90 day') will work..

[0bmay]

Thank you @deluan!

[MichaelBechHansen]

Smart playlists are similar, but not the same as normal playlists in the Subsonic API - yet they are exposed via the same API endpoints. Some differences are, smart playlists are readonly, and are auto regenerated on read (I believe).
This poses some challenges for client apps / API consumers.

For client apps to be able to give a good user experience, information about what playlists are smart (readonly, auto regenerated) and what playlists are user edited is necessary. This information does not seem to be present with the current implementation.

Some points from a discord discussion on smart playlists in the context of the Subsonic API:

Is there any way of distinguishing smart playlists from plain old user-maintained playlists?

• so an app does not offer to add tracks to a smart playlist.
• so an app can know whether it syncs a changed user maintained playlist OR is refreshing a smart playlist.
• ...

Suggestion for one way to add extra information to the Subsonic API:

I think it's pretty safe to add properties to the existing Subsonic JSON/XML response outputs.
So one way could be to add a "navidrome:properties" object to the playlists:
"playlist": [
{
"navidrome:properties": {
"type": "smart", // smart, manual
....
}
"id": "af1a19a2-bafe-4b17-841c-73b0175891a3",
"name": "ABC",
"songCount": 0,
"duration": 0,
"public": false,
"owner": "demo",
"created": "2021-11-28T07:21:40.384665077Z",
"changed": "2021-12-03T09:13:46.098202015Z"
},

...

The presence of the "navidrome:properties" could also be conditional on the client sending some magic parameter in the request.

So in sum, one way to deliver extra playlist information to client apps would be:

• Add an extra, custom parameter to the getPlaylists.view request, for example: extensions=yes. When the server sees extensions=yes, it will output additional, non-standard, properties in the response.
• API consumers will have given the extensions=yes parameter, and be prepared to handle custom properties like "navidrome:properties".
• Legacy client apps will not give this parameter, and will therefore never see anything new.

[aewne]

Would it be possible to add playlist name/id as a selector field for the smart playlists? That way you can mix playlists together without all the hassle that it is now.

[AndersMoberg]

I renamed my NSP files to include emoji, and Navidrome started counting them as separate playlists.. ergo, it wouldn't purge the smart playlists that didn't have a file still.

Green are the new files, yellow are the previous names of these playlists.

[certuna]

Yes, playlist import is a one-direction process unfortunately. Change the name of the m3u/nsp, and it gets re-imported as a new playlist.

[ryoku-cha]

Is there any way to assign the Smart Playlists to any user (or all users)?

I tried putting the following in the .nsp File. But that did not change anything.
"owner": "username"
-> where username was the username for my account.

Currently my Smart Playlists appear only for the "admin" User, but not for my daily use user-account.

[TheWrathKing]

Is there any way to assign the Smart Playlists to any user (or all users)?

Yes, this is possible in the UI. When logged in as admin, click on the icon next to the word "Playlists" on the left side menu and in this new screen you can make the playlists 'public' or edit them.
In the 'Edit' screen you can change the owner of the playlist.
Hope this will help

[deluan]

Would it be possible to add playlist name/id as a selector field for the smart playlists? That way you can mix playlists together without all the hassle that it is now.

@aewne Yes, this is planned.

@MichaelBechHansen, I think a generic property instead of navidrome:properties would be a better approach, that could be used by other server implementations

[RedKage]

Hello, just a heads up while testing v0.47.5, it seems some of my smart playlists NSP are broken.
After a full refresh my playlists got emptied.
They contain NSP like this one

{
  "all": [
    {"is": {"rating": "5"}}
  ],
  "sort": "playcount",
  "order": "desc"
}

Which was working fine on the previous version.

The issue is that the value, in the above 5 is a string with double quotes in my NSP.
I changed my NSP with "rating": 5 without double quotes, touched and refresh and now they all work properly.

It makes sense, the value is a number.

This works as of 0.47.5

{
  "all": [
    {"is": {"rating": 5}}
  ],
  "sort": "playcount",
  "order": "desc"
}

This is not a bug, just an observation.

Also confirmed "rating": 0 works!
Thanks for the new release!

[gardebreak]

I have a "Recently Played" smart playlist:

{
  "all": [
    {"inTheLast": {"lastPlayed": 30}}
  ],
  "sort": "lastPlayed",
  "order": "desc",
  "limit": 50
}

Only tracks that I play through the web UI will show up on the list. It seems any tracks played on an external client don't get marked as "played." Is that the expected behavior?

[deluan]

@RedKage thanks for reporting this. I'll check if I can keep supporting both syntaxes in future releases

@gardebreak, you need to enable "Scrobbling" in your Subsonic client, see #593

[0bmay]

I think the smart play list sorting by random may be broken.. I'm getting back items that are sorted by song title instead..

{
"all": [
{ "any": [
{"is": {"loved": true}},
{"gt": {"rating": 3}}
]},
{"notInTheLast": {"lastPlayed": 180}},
{"contains": {"genre": "metal"}},
],
"sort": "random()",
"limit": 25,
"source": "https://www.reddit.com/r/navidrome/comments/qxh3qu/comment/hl9fb6x/?utm_source=reddit&utm_medium=web2x&context=3"
}

this returns a list of:

#EXTINF:260,Corporate Avenger - $20 Bill
#EXTINF:186,Celtic Frost - (Beyond the) North Winds
#EXTINF:192,Spineshank - (Can't Be) Fixed
#EXTINF:256,The Crown - (I Am) Hell
#EXTINF:275,The Resistance - (I will) Die Alone
#EXTINF:426,Mudvayne - (K)now F(orever)
#EXTINF:340,Pantera - (Reprise) Sandblasted Skin
#EXTINF:185,Judas Priest - (Take These) Chains
#EXTINF:239,Mudvayne - -1
#EXTINF:262,Static-X - ...In a Bag
#EXTINF:192,Rise to Fall - ...Kills the Reason
#EXTINF:550,Seemless - ...Things Fall Apart
#EXTINF:378,Isis - 1,000 Shards
#EXTINF:201,Five Finger Death Punch - 100 Ways to Hate
#EXTINF:386,Megadeth - 1000 Times Goodbye
#EXTINF:374,Megadeth - 1000 Times Goodbye
#EXTINF:386,Megadeth - 1000 Times Goodbye
#EXTINF:232,36 Crazyfists - 11.24.11
#EXTINF:223,Lamb of God - 11th Hour (Remixed) [Remastered]
#EXTINF:350,Megadeth - 13
#EXTINF:396,Corrosion of Conformity - 13 Angels
#EXTINF:252,Black Label Society - 13 Years of Grief
#EXTINF:161,Seance - 13th Moon
#EXTINF:338,Iron Maiden - 2 A.M.
#EXTINF:360,Iron Maiden - 2 Minutes to Midnight

songs that match the criteria, that all start with symbols or numbers, sorted by title. I would expect the listing to be more random. I know there were some changes to the smart lists in the latest release. Did random get broken, or is there a new way to randomize the returned data?

Thank you for all the hard work on navidrome.

[deluan]

Yeah, the fix to avoid possible SQL injections also closed this loophole. I just pushed a fix for random. Now you have to specify "sort": "random" (without ()) and it should work. To test it out and use it, please update to the latest dev build

[0bmay]

Thanks! I run navidrome from a docker image, so I will have to wait for either a dev build or a the next release. I didn't see anything in the documentation on how to build a docker image.

[sandreas]

May I ask / suggest a few things? :-)

• InTheLast is something that could be accomplished via lastplayed? InTheLast feels like it is very special and does not align well with the other filters.
• Would limit as a raw number be enough? I often use the Download feature to copy music for my car... a limit of 8GB instead of 100 Tracks could be helpful
• Is there a filter for MediaType (Music / Podcast)?

As a suggestion for the smart playlist UI I personally like the iTunes interface for this:
https://www.youtube.com/watch?v=Wc_iyQGOgCw
It's not perfect, but I think it is a good compromise between simplicity and flexibility... especially the nesting element with the three dot buttons.

[deluan]

May I ask / suggest a few things? :-)

Thanks for the suggestions!

• InTheLast is something that could be accomplished via lastplayed? InTheLast feels like it is very special and does not align well with the other filters.

lastPlayed (meaning "last time the track was played") is a field, and InTheLast (meaning the field value is in the last N days) is an operator, so they are not interchangeable. InTheLast can be used with other fields, as dateAdded, dateModified and dateLoved. I see the field naming is not consistent, but this is not the final version for this feature.

• Would limit as a raw number be enough? I often use the Download feature to copy music for my car... a limit of 8GB instead of 100 Tracks could be helpful

That's a great suggestion, I'll see if I can implement it in a future version.

• Is there a filter for MediaType (Music / Podcast)?

Not yet, as Navidrome currently treats all files as music. But if you put your files in different folders, you can filter by "contains": {"filepath":"/directory"}

As a suggestion for the smart playlist UI I personally like the iTunes interface for this: https://www.youtube.com/watch?v=Wc_iyQGOgCw It's not perfect, but I think it is a good compromise between simplicity and flexibility... especially the nesting element with the three dot buttons.

Yes, this feature is "inspired" by iTunes' Smart Playlists. And the idea is to add an experience very similar to iTunes in the new UI.

For now, you can try creating Smart Playlists using Feishin

[sandreas]

lastPlayed (meaning "last time the track was played") is a field, and InTheLast (meaning the field value is in the last N days) is an operator, so they are not interchangeable. InTheLast can be used with other fields, as dateAdded, dateModified and dateLoved. I see the field naming is not consistent, but this is not the final version for this feature.

Ah I see. Thank you.

That's a great suggestion, I'll see if I can implement it in a future version.

Ok, nice.

Not yet, as Navidrome currently treats all files as music. But if you put your files in different folders, you can filter by "contains": {"filepath":"/directory"}

That's a nice workaround. Great.

Yes, this feature is "inspired" by iTunes' Smart Playlists. And the idea is to add an experience very similar to iTunes in the new UI. For now, you can try creating Smart Playlists using Feishin

After playing around a bit, I'm not sure I'll ever need a GUI. It's so easy to edit the JSON files and I'm familiar with it, so I really like this feature.

The only thing I really was struggling with: JSON Parser error just skips the playlist and it is never reported in the WebUI anywhere. I would expect at least a small warning or something. I thought it did not find anything and had to open the docker logs.

I opened a feature "smart-tags". This would be a perfect little addition for smart playlists and you could auto-tag a lot of scanned files with some things.

Thank you for making this tool. It's just great.

[vurtomatic]

@deluan For sorting, is it possible to sort by Album, and then further sort by Date Added?

I also noticed the album tracks displayed aren't sorted, I can't tell what order they're in.

[calm3285]

I'm trying to make a playlist which contains all of my favorited albums.

Issue is, this playlist is just a list of the two songs I have favorited as opposed to the 600 something albums I'm expecting. Can this be done?

@qouesm Did you find a solution?

[tbleiker]

As far as I understand, the query defined by a smart playlist depends on the owner's stats. I would like to define a smart playlist (e.g. Favorites of a specific genre) once and every user can use the playlist based on his/her own ratings. Currently - to the best of my understanding - I have to create such a smart playlist for each user (which obviously does not scale well... 😄).

Is there anything planned with respect to this? @deluan Maybe this feature can be added to the todo list at the top?

[sammypanda]

What is the 'title' property intended for?

{
  "title": "Recently Added",
  "comment": "All tracks above 3 stars.",
  "all": [{
    "inTheLast": {
      "dateadded": "90"
    }
  }],
  "sort": "dateadded",
  "order": "asc",
  "limit": 25
}

When using the title property like this, the title of the playlist for me is not updated. Is title not meant to be able to change the playlist title? (don't worry that the comment is unrelated it's just there as a placeholder).

[SickProdigy]

What is the 'title' property intended for?

{

  "title": "Recently Added",

  "comment": "All tracks above 3 stars.",

  "all": [{

    "inTheLast": {

      "dateadded": "90"

    }

  }],

  "sort": "dateadded",

  "order": "asc",

  "limit": 25

}

When using the title property like this, the title of the playlist for me is not updated. Is title not meant to be able to change the playlist title? (don't worry that the comment is unrelated it's just there as a placeholder).

It may be later added, but as of now the title of the file is the title taken in my experience.

[deluan]

Is there anything planned with respect to this? @deluan Maybe this feature can be added to the todo list

@tbleiker Yes, this is planned when we get the Smart Playlist UI in place

@sammypanda As @SickProdigy mentioned, title (and comment) are not valid attributes, but I plan to add them and more in the future, once I'm back to finish this feature.

Actually they are, see the first example in the description of the issue. Title is actually name.

[sammypanda]

Is there anything planned with respect to this? @deluan Maybe this feature can be added to the todo list

@tbleiker Yes, this is planned when we get the Smart Playlist UI in place

@sammypanda As @SickProdigy mentioned, title (and comment) are not valid attributes, but I plan to add them and more in the future, once I'm back to finish this feature.

okie! :)

weird, the comment one seemed to work; maybe I forgot and did it a different way 🤔

[RyanVerhey]

Is there a way to add comments to the NSP files? Like // Comment, # Comment, etc.? I tried using bogus attributes, like "_name", but the parser doesn't like that very much. I have a playlist that omits tracks in some other playlists, but I'd like to be able to write down what those playlists are vs. having to look up the UUIDs each time. Thanks!

[sandreas]

Is there a way to add comments to the NSP files?

@RyanVerhey

I don't see that this could be possible. JSON is a specified format that does not support comments in general. The way you tried is the only possible way (adding custom props) to do something LIKE comments and if this leads to an error, I don't think it's possible.

I personally use JavaScript-Files to store my playlists with comments and do copy and paste if I refresh them.

[djjudas21]

Is there a way to add comments to the NSP files?

The next best thing I can think of would be to have a text file adjacent to the playlist, e.g. playlist.nsp and playlist.txt and then keep your notes in the text file.

[sammypanda]

Is there a way to add comments to the NSP files?

@RyanVerhey

I don't see that this could be possible. JSON is a specified format that does not support comments in general. The way you tried is the only possible way (adding custom props) to do something LIKE comments and if this leads to an error, I don't think it's possible.

I personally use JavaScript-Files to store my playlists with comments and do copy and paste if I refresh them.

it could be if nsp was made to be jsonc instead of json

[RyanVerhey]

Thanks for the responses, everyone. I looked into JSONC or JSON5, but there don't seem to be any good available packages that can read the .nsp file the way Navidrome uses the standard JSON library, i.e. reading from a file stream. JSONC/5 don't seem to be well supported in the Go ecosystem, as far as I can tell.

[deluan]

Thanks for the responses, everyone. I looked into JSONC or JSON5, but there don't seem to be any good available packages that can read the .nsp file the way Navidrome uses the standard JSON library, i.e. reading from a file stream. JSONC/5 don't seem to be well supported in the Go ecosystem, as far as I can tell.

Maybe changing the format of the file to YAML? As YAML is a superset of JSON, it would still read .nsp written in JSON. I'll try this approach and see if it is feasible and address the concern here.

[sandreas]

Maybe changing the format of the file to YAML? As YAML is a superset of JSON, it would still read .nsp written in JSON. I'll try this approach and see if it is feasible and address the concern here.

Please don't use YAML. As I remember this feature is targeted as a UI editable smart playlist. I would not invest more time into changing the format or support features that are ONLY made just to be able to manually edit text files. The comments are only required if there is NOT a UI with descriptive elements.

[RyanVerhey]

That’s a good point. If manual file editing is supposed to be temporary, then there’s no point needlessly jumping through hoops to get comments to work.

[0bmay]

I know it's not part of the file, but you can add comments to the generated playlist in navidrome for the nsp files.

[sandreas]

I think there is no need for changes. Comments are already supported on the playlist level. The most i would do is to add a comment property on the rule level, bot only if this can be integrated easily...

[deluan]

@0bmay

I know it's not part of the file, but you can add comments to the generated playlist in navidrome for the nsp files.

You can set the Name and Comment displayed in the UI for a NSP file:

{
  "name": "Recently Played",
  "comment": "Recently played tracks",
  "all": [
    {"inTheLast": {"lastPlayed": 30}}
  ],
  "sort": "lastPlayed",
  "order": "desc",
  "limit": 100
}

[deluan]

@sammypanda

What is the 'title' property intended for?

title is used as a property in the expressions, it is supposed to match the title of a song. It is not used as the Name of the playlist. What you want is name. See the comment above.

[deluan]

Is there a way to add comments to the NSP files? Like // Comment, # Comment, etc.? I tried using bogus attributes, like "_name", but the parser doesn't like that very much. I have a playlist that omits tracks in some other playlists, but I'd like to be able to write down what those playlists are vs. having to look up the UUIDs each time. Thanks!

I just pushed a change (7871d69) that enables comments in JSON. Should be available soon in develop builds. It will be part of next release (in the next few days)

[RyanVerhey]

Is there a way to add comments to the NSP files? Like // Comment, # Comment, etc.? I tried using bogus attributes, like "_name", but the parser doesn't like that very much. I have a playlist that omits tracks in some other playlists, but I'd like to be able to write down what those playlists are vs. having to look up the UUIDs each time. Thanks!

I just pushed a change (7871d69) that enables comments in JSON. Should be available soon in develop builds. It will be part of next release (in the next few days)

Thank you! That’s awesome, and just stripping comments vs. adding support for JSON5/C/YAML seems like a much simpler change.

[sammypanda]

@sammypanda

What is the 'title' property intended for?

title is used as a property in the expressions, it is supposed to match the title of a song. It is not used as the Name of the playlist. What you want is name. See the comment above.

thank you :)

[sprnza]

Is there an update on this? (Last Played for all users)

[dathbe]

(Last Played for all users)

I’m not necessarily saying this can’t be implemented properly, but I do want to point out that there are some real privacy issues raised by it.