All planned/requested features #13
Replies: 5 comments 14 replies
-
|
I use Nodezator to visually build complex analytic SQL queries from library of simple, parametrizable ones (nodes). I also have various output nodes for data visualization. Often used queries can be then exported as Python code for more convenient use. All works great, but I really struggle with Nodezator's nodepack machinery. The nodepack is basically a Python package, and nodes in generated Python script are imported from it in usual way. But it's not possible to specify package to be used as nodepack, only as directory. This has several annoying consequences:
So, It would be really great if method for importing nodepacks would be changed to use modules instead current one (directory). The implementation could be really simple. Instead select directory dialog, you use simple from with string entry to get module name, import it and then use module. path attribute to get it's location to scan it for contained categories and nodes. This would also allow deeper nesting to organize nodepacks, for example: org.sql.category.node One can then import "org.sql" to get all contained categories and nodes from it without other two. |
Beta Was this translation helpful? Give feedback.
-
|
I understand your reasons for directory approach, but in fact you can keep it "as is" but using normal packages all along. User "playground" nodepack directory could be integrated into Python site-packages using .pth file (see https://docs.python.org/3.8/library/site.html or https://stackoverflow.com/questions/15208615/using-pth-files), so you can use them like they would be installed via package manager. This method is for example used when you install local package via pip in developer mode. A side note: Is it really necessary to "pretty-format" imports in generated script? Is really annoying and against source formatting rules. I had to disable format_local_imports in graphman/pythonrepr.py every time I pull nodezator from github. |
Beta Was this translation helpful? Give feedback.
-
I'll take this into account on my analysis. As long as it is easy and straightforward for new users, it should make into Nodezator without problems. As I said, the final implementation will still undergo some analysis and discussion. I also need to read about all those alternatives to make sure we are on the same page, which is why I need the extra week or so to organize my thoughts and plan a first draft of the implementation to be presented for discussion.
No, it is not necessary. In hindsight, this was a bit of an overstep on my part. I mean, though I do like the formatting because it leaves the names of the packs/categories/nodes visually aligned and thus, in my opinion, easier to scan with the eyes, I should not have made this decision purely based on my opinion. This is even more true regarding formatting, since it is too subjective and reliant on conventions rather than logic. I'll remove it. Since it is a really small change, there's really no need to schedule it, I'll just remove it at the first opportunity. |
Beta Was this translation helpful? Give feedback.
-
|
When we're talking about nodepacks, the current method for their selection is really inconvenient because:
I'd like propose different nodepack managemen:
BTW, is it really necessary that file dialog always opens in home directory? Wouldn't be current working directory better? Or directory where currently loaded .ndz file is located? Or at least, keep the last used directory between dialog activations, so users don't need to navigate from home to the directory they actually use for the project every time the file dialog opens. |
Beta Was this translation helpful? Give feedback.
-
I agree. The current node pack management has many drawbacks, like the ones you just listed regarding the need to select node packs again for each new .ndz file created and the inability to select node packs from different directories. I've been already aware of some of those drawbacks for a long time. I just didn't fix those problems before because I prioritized fixing more critical bugs and implementing more important features so that I could publish Nodezator early and fix such issues and implementing other features shortly after the release, as I am doing now.
Thank you for such detailed proposal. I'll take your points into account (not only the ones in this last message, but the previous ones as well) in the draft I'll publish for discussion like I said before. I believe most if not all your points will make into the final draft.
Yep, I agree, I myself find this file manager behaviour inadequate and have been wanting to change it for a long time, but as I said I've usually invested my time on more critical stuff. Of course now that Nodezator is released I'll be finally able to give attention to this kind of problem as well. Thanks again for such detailed account of your problems with Nodezator and the related proposals. They are always welcome. The more info I can gather about this the better, as it helps define a clear picture of the problem and the possible solutions. As I said before, I just need some time to organize, plan and present the changes for discussion. I also have to tend to my current development tasks, like the changes already scheduled for implementation. I just updated the pillow example node pack so the viewer nodes can now handle window resizing and also scrolling and dragging images on the screen when they are larger than the window. I also recently came up with a solution on my mind to add branching to Nodezator (nodes representing if/elif/else blocks as well as pattern matching), including how to translate it into plain Python for the Python exporting feature. This is yet another set of changes for which I'll need to plan and write a draft and present it for discussion. |
Beta Was this translation helpful? Give feedback.
-
|
I forgot to remove the format_local_imports() function on my recent upload to pypi, but I just removed it in the branch with the changes I'm currently working on, you can see the commit here (52209e2). So the change should be available pretty soon on pypi when I merge the branch and upload to pypi. As soon as I'm finished with my current tasks I'll tend to the new node management approach. |
Beta Was this translation helpful? Give feedback.
-
|
First off I want to thank you for all you have done making an amazing program. I was looking for something node based and all the other programs I had come across seemed to be specialized and not focused on a generalized approach. I just want to thank you again for starting this project, it definitely fills a need. My brother and I have attempted to create code in it and import a few functions, and came across some possible changes you may want to consider. We have cloned from github and are looking at how feasible it is to experiment with some ideas and test them. We dont have a lot of time to commit to it on a scheduled basis, however we would not be opposed to discussing ideas and suggestions and maybe make minor contributions to the project if we can. The following is a summarization of somethings we had discussed would greatly open up functionality and ease of use, these are just some suggestions we would like to offer and are considering trying to figure out ourselves. Perhaps you have already addressed some of this. We had even created a very simplified bridging node before we noticed you were working on what seems to be a much finer model, so you may have once again already have it on your list to address. If a nodezator graph could include either input nodes or an option to change data nodes to an input type, then when you export to python those input nodes could be placed as function parameters. Similarly, if a nodezator graph could include an output node, variables passed to it could be returned from the exported python code. This prevents you from having to do any manual code changes to your exported graphs. If ndz files included inputs and outputs, then they could be imported as nodes using the code they automatically generate when they are exported as python code directly. Using only the ndz files to build all your nodes and drop existing graphs into new graphs as nodes from ndz files would allow you to build nested functionality very similar to the way blueprint uses functions and macros. To take it a step further, it would be very flexible if you could double click a ndz file based node or right click any node that was imported from an ndz file and that would open the graph for editing the ndz file. If all ndz file based or python based nodes could be compared to the copy saved in a graph and then they just turn red or have some other form of visual change to denote that they have to be refreshed, this would allow you to make changes to nested ndz file nodes and then save them and refresh the higher level graphs that use them. This would also fix the issue of having to remove and re add libraries if you change only one node within a library. |
Beta Was this translation helpful? Give feedback.
-
|
I appreciate and welcome the time you and your brother already committed and are wiling to commit to this project. Don't worry about whether something was already suggested/tried or not. It would be too time consuming for anyone to read all content already produced in the discord and discussions. In my honest opinion, when someone suggests something that was already tried/suggested it is a good thing, because it means that person has good insight on possibilities/opportunities presented by the app. I just gave your message a quick read. I'll get back to you on your suggestions on Monday. Pardon me for not addressing your suggestions right away. I'd like to do it as soon as possible, but I'm just off from a day full of work and I need the Sundays off so I can replenish my energy to work at my best the rest of the week. Additionally, your suggestions touch a very interesting topic that I'd like to take my time to write. By the way, I do actually dedicate a little time even on Sundays just to check the discord and github to see if an user has some urgent issue that warrants my attention like a critical bug in something they need right away for an urgent task, things like that, in order to not leave any user unattended in a dire situation. So, if you ever have something urgent even on Sundays, don't hesitate to reach out for help. Have a nice weekend, see you on Monday. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the kind words. There is no rush on our end for you to review things, we just honestly see a good future for it where we might gravitate to using it as a major tool for programming and would like to work towards it being something in that direction. We will keep you updated as we can on our end to see if insight can be swapped. Just don't feel like you need to rush things, we do not expect you too unless you yourself are just that excited to pursue it. But we are happy to hear your thoughts on it when you are ready. |
Beta Was this translation helpful? Give feedback.
-
|
Here are my thoughts on your message, point by point (this is part 1 of 2; I will write part 2 of 2 tomorrow):
I'm glad the app meets your needs. There's still tons of stuff to improve and to implement, of course, but we are advancing, slow and steady. Other users have similarly described how Nodezator helps solve their needs of a generalized approach for visual programming in Python. That makes me specially glad not because I think Nodezator is superior to other existing Python node editors, but because it clearly addresses problems for which other editors are not suitable. This is why I made it, even though there were already other great editor at the time. I just needed something that I didn't find elsewhere, that's all.
Again, thank you for willing to take these extra steps, however small they are. Take your time and do not hesitate to share ideas. Due to how complex a node editor app is, sharing knowledge/discussing/giving feedback are essential to ensure an accurate understanding of problems and possibilities for the app and cooperation. As the open-source maintainer of the app, I also own the community the publishing of a proper CONTRIBUTING.md file in the repository, explaining the ins and outs of the package, how it works, how it is organized, strengths, weaknesses, best approaches for contributions and many other useful stuff. Sorry for not having released something like that at the present time. It is just that since the release of the app this past June I've been implementing new stuff non-stop. Thankfully, it has been a healthy rhythm of "new feature" work, but nonetheless it has hindered other equally important work like better guides for people willing to contribute. This is totally on my radar though.
Do you mean "branching" nodes? If so, the current task being handled is indeed the implementation of branching and looping in Nodezator via new special nodes. The task is in the design phase, but I should soon start implementing it after finishing the design and waiting a bit for user to review and give feedback. The work is being discussed here and the design draft is published and being updated here. |
Beta Was this translation helpful? Give feedback.
-
|
Ah yes branching, sorry I was tired at the time i typed it up and accidentally put bridging. |
Beta Was this translation helpful? Give feedback.
-
|
Here's part 2 of 2:
The comments above show that you already have good insight on some exciting possibilities for Nodezator. I'm all for the inclusion of such special input and output nodes in Nodezator as long as we achieve a suitable design for the feature. As you described, being able to define inputs and outputs inside Nodezator will help us not only to export the .ndz files as functions with defined signatures but would also allow said functions to be used as nested graphs within other .ndz files, just like group nodes in Blender or macros in Blueprint as you said. The idea have been mentioned before more or less 6 months ago by asgore in this highlighted youtube comment, where he asks "can we create new functions in the editor using nodes?". At the time, my answer was that I planned to use the exported .ndz files as bases for new "group nodes" and that 99% of the work would be done in the app, only 1% would be quick small adjustments in the exported function. (I also mentioned that I'd implement it a little bit after the release, but since then other features of higher priority have been popping up; but that's ok, it is just how open-source projects are; my ideas about a lot of feature have changed a lot since then as well). Coming back to your suggestion, I'd much prefer if we could create and edit the new functions/group nodes 100% inside Nodezator as you said yourself. And more, the idea not only crossed my mind, but I actually already attempted implementing it with some degree of success. A few months before even having announced the existence of the app (before March 2022) I tried a design I came up with, but despite said degree of success there were some problems for which I couldn't find solutions that satisfied me 100%. In summary, when comparing the gains with the costs (the infrastructure needed to support the feature and the resulting maintenance that would be required), I decided to abandon that path of development. However, I didn't give up on the idea. Now that the project was released and there are more heads thinking, we might actually be able to achieve a satisfactory design. However, such change is a big one, and as such, it requires time and the proper procedure. Just like the branching and looping feature for Nodezator, that required a discussion, a draft explaining the design (which I'm almost finishing) and will require a month or more to actually implement, this suggestion of yours will need a similar procedure. Considering there were other users that mentioned input/output nodes as well and I do think it is something that is desirable and can be useful, I think at this point we can consider it a planned/requested feature already. I'll list it in the table at the top of this discussion that contains the feature requests. After this, though, please keep in mind that it will take some time until we can schedule, draft/design, discuss and finally implement this feature (provided we achieve a good design, which I think is possible), since we already have a lot on our plate. And I'm not even talking about the current task of implementing branching/looping, but the other listed ones as well. Once I'm able to give attention and time to this feature though, I'll share my thoughts as well as incorporate your ideas and the ideas of other users as well. Ideally, I'd love to have people that understood enough the inner workings of Nodezator to be able to this kind of tasks in parallel so that we could merge, but even if we had such people I think it still would not be possible because despite its more or less 04 years in the make Nodezator still has a lot to mature until a definite API/shape is achieved so that more stable and coordinated development can be done. As Nodezator is now, it is still changing a lot and each feature added often represents a new revision and refactoring in several subsystems. Splitting the development effort is still something to be achieved. But slow and steady, I believe we'll get there. Just know, however, that I'm commited to the community and this project and also to share my knowledge as the time goes, so that more and more people can better understand the app and contribute as well.
It is impressive to see that you gave the feature this much thought. When the time comes, as I described, we'll be able to discuss all of those points better and in detail, and to incorporate more ideas into the design. Until there, I think you and your brother are already in a great path working in your clone and maturing the ideas you already presented. Also, as always, do not hesitate to ask if you have any questions or to share more ideas and suggestions if you have those. Peace. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @KennedyRichard, @ambientdark333 and I have been working on a proof-of-concept for input and output nodes and it is far from finished but we have a working example to share and would like to get your thoughts. The closest construct we found for input nodes in the existing code was the following widgets: We needed a way to separate the input only nodes from standard nodes so we added new variable to each of these called: node_io_type When creating the node instance, the class name is referenced so to create a different type we subclassed each of the three classes listed above and called them The only differences between the subclasses and originals is the 'name' variable and the value of the new 'node_io_type' variable. Eg. class InputStringEntry(StringEntry): These proxy nodes work great as inputs except for the fact that they have an input connection circle and a red x that allows the user to change the node variable type. This will need to be disabled in order to continue using this type but we haven’t figured that part out yet. For outputs, the closest existing construct we could find that would work well with the other existing systems is the dict. We added a new entry under builtin constants for a dictionary type called dictOut(**kwargs) along with its function _dictOut1. We added widget presets for the new input and output types under: nodezator/graphman/presets Next, under editing/objinsert, we added a new key and value to the node_data dict called node_IO_type with a default value of ‘IO’. When the node_hint is a string (which is the case when we insert dictOut b/c it's a builtin) we can determine the node is an output by looking through the node hint for specific text i.e. dictOut. Once we determine that the node was an output type we change the value of node_data[‘widget_IO_type] to ‘output’. Similarly, when inserting proxy nodes we can look for the text ‘input’ under the class variable using: if "input" in node_data.get('widget_data', {}).get('widget_name') and set the node_IO_type value to ‘input’. Finally, to use these new input and output nodes in exported code, we revised nodezator/graphman/pythonrepr as follows: Added global variables FUNCTION_INPUTS and FUNCTION_OUTPUTS. Similarly, under callable_node_to_text if we find that the current node is an output node, we add the appropriate items to FUNCTION_OUTPUTS for each subargument instead of processing the node text. The repr format uses type hints and default values based on the input nodes when writing the function text so that it matches the format required to turn this code back into a node. The next logical step (after fixing the existing issues) would probably be to set up a menu entry to export/save your ndz file to a node automatically so the folder structure, naming, and node pack importing can be done for you. Let me know what format you would prefer we use to share the code in progress such as a git branch or fork or just text or something else. Also please let me know if there is a different thread or location you would prefer to discuss the implementation details and questions outside of this requested features post. See attached for some images of this proof-of-concept code change. Known issues to address: Let us know what you think and how you would prefer that we go about collaborating on this project further. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @sbudro, just stopping by to acknowledge the receipt of your message. I'll give both your message and your proof of concept the attention they deserve and then share my thoughts on them after I have some time to ponder. I'll probably give a proper answer by Tuesday. Thank you for your precious time and effort. Have a nice weekend! |
Beta Was this translation helpful? Give feedback.
-
|
Hello, Mr. @sbudro I've read and reread the implementation and proof of concept both you and @ambientdark333 provided. First of all, the manipulation of the different modules to achieve your design shows a lot of discipline, focus and attention to detail. It is also impressive how acquainted with the source code you both got. Congratulations on all that, really. As I told ambientdark333 before, in previous replies,
This just means guaranteeing as much as possible that the design...
Thankfully, even though there's still a lot to be discussed, you clearly have not lost sight of this. For instance, you went as far as considering the relationship between the new nodes and the python exporting feature. This shows a lot of attention to detail. Even so, as I also pointed out before, a feature such as this one still needs to be subject of discussion, analysis and feedback from the community, not only myself. I created a discussion dedicated to this topic: #37 I added the input/output nodes to the list of requested features in the top of this discussion. I apologize for not doing this earlier, as I said I would before:
I also put a link to the dedicated discussion there. I'll share my thoughts on your proposal in that discussion. It will take some days, though, because of other tasks I'm doing. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @KennedyRichard, it looks like you already have most of the common node editor features on your list! I have a few more feature ideas that I don't see on your list yet.
That's all for now. None of these ideas are high priority for me personally--I just wanted to write them down because I didn't see them listed above. As you may have guessed, the feature I'm dying for right now is higher-order functions/callable nodes :) Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
First of all, thanks for the feedback/requests. This reply is to confirm I'm aware of your message and will write a proper reply as soon as possible. ETA is a couple of days. Peace |
Beta Was this translation helpful? Give feedback.
-
I never heard of such feature before. I'm glad you brought it to my attention since looking into the page you linked and researching the feature a bit more showed me that such feature and variations of it seem to appear in a number of visual editors, not just Nuke. To be frank, only after researching this feature a bit more I could see its usefulness. At first, though sold on the feature, my concerns were to which extent and at what cost the feature would be useful. "To which extent" because the app already has redirect nodes to ensure consistent values are used where needed, that is, a single value on the graph can be propagated to anywhere else in the graph as needed by extending the connection to where it is needed using the redirect nodes. This works quite differently from cloned nodes, but serves the same purpose. However, it is true that it results in extra wires in the graph. Each new node where the values are used would require more and more connections and redirect nodes between them. Nevertheless, the differences don't stop there. Cloned nodes have the additional advantage of affecting the entire node (that is, the arguments used on them, not the incoming connections), so the idea behind it, as you pointed out, is to replicate a specific configuration of a node, like the 3D transforms example you gave. So there's no doubt this sets the feature apart. Redirect nodes still have their usage and advantages over cloned nodes, but in fact the features are not interchangeable and node cloning/decloning can hold its own as an unique feature. Moreover, the "at what cost" part was in regard to changes needed in the backend to allow such feature, including not only its own implementation, but possible additional changes elsewhere on the app to allow for its existence in harmony with other feature. However, after thinking about it for some time I was able to come up with a design simple enough and solutions for the major challenges that the implementation would entail. There's still some loose ends to sort out, though, but minor ones, thankfully. I'll give you an example. Nodezator has a feature to export the graph to Python code. So when exporting cloned nodes to Python code, should I just copy the values of the cloned nodes in the calls, or should I create the values as independent variables and only reference them in the function calls? I'll probably go with the latter option, since it avoids needless repetition and highlights the relation between the calls/values. I won't go further in my analysis because it would require too much time and an entire discussion of its own. In the end, however, my conclusion, after careful consideration, is that the feature is indeed useful, unique and its implementation can be relatively smooth, thus I added node cloning/decloning to the list of planned/requested features, as you can see in the table at the top message of the discussion. As a curiosity, it find it interesting how node cloning seems to share a lot of similarities with group nodes (like group nodes in Blender). I mean, group nodes keep a combination of nodes consistent and allow the users to change only the values and incoming connections as they desire, whereas node clones use a single kind of node, but keep the values consistent across the clones so that, except for the incoming connections, the different nodes share the same values and thus their configuration is kept consistent in their own way.
I thank you for the extra effort of trying to provide a simpler alternative to cloned nodes. However, as you pointed out, it seems a feature like this has not been put to use for good reason. Specifically, I think the problem with this design is that since the wires represent crucial information about the travel of data in the graph, hiding them is harmful to understanding/using the graph. The implementation requires little effort, but the usability cost is too high. Even so, thank you for providing the alternative, really. All ideas are welcome because even the more controversial ones may help us broaden our perspective on problems.
I'm actually surprised I didn't add backdrops to the list of features before. I just did it now. However, I think text blocks and backdrops should be kept as separate objects, they do have different purposes as well, in my opinion. Backdrops are great to group nodes together and indicate their purpose with the text in their header. Text blocks are extra context given by the user and they are equivalent of code comments in Python scripts (this is precisely what they become when the graph is exported to Python code). With that in mind, and assuming you agree with me on what I said up to this point (it's fine if you don't agree, just let me know so we can discuss this further), do you still think it would be desirable to have the ability to change the color of the text blocks? Since they are just comments, I think it is fine for them to have a single color. Only the backdrops should have their color picked by users as they desire. Text blocks should be resizable though, as backdrops naturally are. About drawing text blocks underneath nodes/wires, I think it would do more harm than good, because whereas the direction of a connection can be inferred even when a text block is covering it a bit (connections are usually longer than the text blocks, so text blocks rarely cover a connection completely), having the text on a text block covered by a connections can be annoying/ugly and might even prevent the user from reading the text underneath. Did you suggest this because of your intention of using text blocks as backdrops? Because, if you did, than you don't need to worry about this at all, since backdrops are something we'll implement anyway. Nonetheless, if you still think this is desirable somehow, just let me know and give me more context into why you still think this can be better, so we can discuss this further. Another curiosity: text blocks have an additional hidden feature (that are not really hidden, it is just that I didn't have time to present it in the manual yet). When exporting the graph to Python code, they are actually positioned in the script just above the node (that is, the call) or nodes closest to it. You see in detail when I have the time to explain it in the manual. When converting backdrops to Python, I think I'll do something similar, but generate a comment similar to a header on top of the exported, with the text used on the header of the backdrop. Yet another: when exporting the graph to Python, the app is aware of which nodes form isolated clusters based on the proximity between them. I could also use this info to implement a feature to generate backdrops automatically based on the proximity of the nodes. But this is just my mind going wild, I'll think about this more carefully when it is time to implement the backdrops.
Yep, I concur. Added to the list of requested/planned features as well. As you said, it is something that will take some time before I can start thinking about it, because it will affect the other features, so I'm focusing in implement them first, then I'll think about how to integrate a timeline.
Thank you for your suggestions/requests and the accompanying links providing more info.
Me too! This is one I'm not only excited to implement, but in need of it as well. When using Nodezator I've being doing some hacks/workarounds in a context where the ability to use the node's callable in combination with a map/filter node would solve the problem right away. |
Beta Was this translation helpful? Give feedback.
-
All planned/requested features
Hello, everyone, the sole purpose of this pinned discussion is to list/keep track of all planned/requested features for Nodezator. I'll be editing it regularly to add new items and edit existing ones.
Check the instructions after the tables below to learn how to request features and how I go about implementing those features.
Also just cause there may be a lot of time between the implementation of the features, doesn't mean there wasn't much work being done during that interval. It is just that some features require a lot of time and effort (redesigning the app, testing, fine-tuning, etc.) and my work on the project also consists of many other things like curating the discussions, researching, answering and solving issues/fixing bugs, producing content, updating the websites/social networks/channels, promoting the project online, etc.
If there is anything you discussed with me but I forgot to include here or in the sublists below, please, let me know!
Other minor features that should be quick or easy to implement and might be added to Nodezator any time:
Other topics for research that may or may not become a planned feature:
Other requested/planned features will only be available for implementation in the future, once the pygame/OpenGL graphics backend for Nodezator is implemented, but they'll be listed below so we can keep track of them;
how to request new features
If you want to request new features, comment below or start a new discussion requesting the feature and I'll list it here depending on feasibility, desirability, usefulness, urgency and other factors.
The more data you provide, the higher the chances of your feature to be listed here, so maybe starting a new discussion with explanations, examples, images, etc. may be a better idea, but don't worry too much about that, just be honest and open to discuss and all should go more than well.
how features are implemented
Every time I'm about to work on a new branch, I'll visit this discussion, pick some items to implement and create a discussion to let you share your thoughts on the ones I pick to implement.
Beta Was this translation helpful? Give feedback.
All reactions