Netdata agent package name conflicting with EPEL package.

[iigorkarpov]

The name of the netdata agent package is the same as the package name used in EPEL. Given that EPEL is an extremely popular repository, this overlap is fraught with serious problems for inattentive/inexperienced administrators. Routine updates of packages in the system may (depending on the current versions) lead to replacing one build with another. It is highly probable that this will cause the package to stop working.
We probably cannot influence the creators of the package for EPEL. In that case, we should consider changing the name of our own package.

# yum list | grep netdata netdata.x86_64 1.33.1-1.el7 @netdata netdata-repo.noarch 1-1 @/downloads9IDpg netdata.x86_64 1.33.1-2.el7 epel netdata-conf.noarch 1.33.1-2.el7 epel netdata-data.noarch 1.33.1-2.el7 epel netdata-debuginfo.x86_64 1.29.3-1.el7 netdata netdata-freeipmi.x86_64 1.33.1-2.el7 epel netdata-plugin-freeipmi.x86_64 1.33.1-1.el7 netdata netdata-repo-edge.noarch 1-1 netdata

[Ferroin]

Alternatively, it looks like DNF and YUM support setting repository priorities (requiring a plugin in the case of YUM), which override version number checks.

At the time when I was creating the repository configuration packages, I had not know about this, so we don’t actually use it currently, but setting our own repository priority to a sufficiently low value such as 50 (default is 99) should resolve this without needing to shuffle package names.

[iigorkarpov]

We can. But if I were the user I wouldn't like this a lot. This could break (theoretically) existing order of handling my packages. If I have dozens and hundreds of servers, this is the last thing I'd like to install. It is good for the new servers but not for existing infrastructure.

[iigorkarpov]

I do not mean this is urgent, but this may cause bad user's experience. As far as I can see, EPEL version is usually lower than ours, so this problem occurs quite rarely.

[Ferroin]

If it actually breaks the existing selection of packages, then the system was already broken. The only case that should happen is if the user was already using repos that had priority information in their configuration, but did not have the plugin installed, which should not be the case on any properly configured system (any repos that have priorities should have pulled in the plugin when they were installed).

Furthermore, it’s actually only needed on CentOS 7 (and/or RHEL 7). DNF provides this functionality natively.

[iigorkarpov]

I started adding tasks to the Ansible playbook to set priorities for yum and I feel like we are opening a door to hell. To begin with, the priorities are clearly not set in either Centos 7 or Rocky 8. The only way to add them to the EPEL repository configuration files with ansible is to overwrite those files entirely with either pre-made files or those generated from templates. The reason is the structure of these files. Each of them contains several blocks, in each of which you need to add a line with a priority. The lineinfile module is not applicable in this situation. You can use the more general replace module, but it requires binding to a regular expression, which makes the solution unreliable. In the end, we don't know exactly how the configuration file was modified.
That is, we are again grossly interfering with the settings of the target server.
So I decided to install the dependencies before the netdata agent package and then install the package with the disablerepo option. Just in case, I also install yum-plugin-priorities, but I don't use it. At the moment this seems to me to be the best possible solution for Ansible.

And the cherry on the cake:

_7. A Cautionary Note.

Note: The upstream maintainer of yum, Seth Vidal, had the following to say about 'yum priorities' in September 2009:

Gosh, I hope people do not set up yum priorities. There are so many things about
priorities that make me cringe all over. It could just be that it reminds me of
apt 'pinning' and that makes me want to hurl.
This matter was discussed in more depth in the mailing list thread starting here. The Repositories article noted in that thread, which discusses the exclude and includepkg options for yum, is a better place to start in understanding priorities.

The primary concern is that priorities is heavily handled over removing packages from the transaction set. It makes it difficult to readily determine what packages are being ignored and why. Even so, it is very flexible and can be extremely useful to provide the largest available list of packages._

[Ferroin]

To clarify my stance on this, I’m not saying priorities are a good option, I’m saying they’re the least objectionable option here.

Renaming the package has potential implications on our marketing and branding, as well as adding a nontrivial long-term maintenance burden for us to deal with and it does not actually solve the problem on existing installs. Most distros are not like Gentoo (where you can actually rename an ebuild and migrate users to the new name seamlessly), you need to provide a compatibility package to transition users over, but because our package is ‘manually’ installed, we can’t get rid of that compatibility package (because removing it would result in the actual package being removed since nothing would be depending on it anymore), and said compatibility package will need to have the same name as our current package, meaning that it will still clash with EPEL.

Excluding the Netdata packages from the EPEL repo at the transactional level persistently requires EPEL to actually be installed when we install, will break if they ever reinstall or update the EPEL repo config package, and requires us to manipulate repo config in ways that are nontrivial to do from a shell script.

Using repo priorities happens to just work in a majority of cases (note that there is a default priority for any repos that do not list it), and the cases where it is likely to break things are exceedingly unlikely.

[Ferroin]

Formally documenting the options here, along with the pros and cons I see, in preparation for asking the product team for an opinion on this in the hopes of resolving the ideological deadlock on this...

Rename the package

Most likely new name would be netdata-agent.

Pros

• Avoids the issue with EPEL in an easily understandable manner for new installs.
• Avoids any extra modifications to user systems.

Cons

• Requires us to keep a netdata package for all of our currently supported distros that depends on the newly named package so that existing installs continue to work. This stems from the fact that we cannot explicitly mark the newly named package as manually installed without violating the requirement to not make other modifications to user systems (DNF/YUM have no way to transparently rename packages while maintaining state). This has a number of problematic implications:
  • It means the issue with EPEL is still an issue for any existing installs.
  • It adds a nontrivial maintenance overhead for us. We have to actually set the package up, keep publishing it for new releases and nightly builds, make sure it doesn’t break, and make sure it only gets published on the correct distros.
  • It runs the risk of confusing users, because they will not readily be able to differentiate between the two package names.
• Impacts all users, whether they were affected by the issue or not.

Set a high repository priority for our package repositories

Pros

• Can be done 100% transparently to users. IOW, they will not see any visible changes unless they were actually affected by this issue.
• Will work out of the box on a vast majority of systems systems with no significant changes.
• Zero maintenance overhead for us.

Cons

• Requires adding a new package on CentOS 7 systems, because YUM requires a plugin to handle repository priorities. Note that we already require additional packages for repo setup on all DEB systems, so this is not hugely out of line for us.
• May result in unexpected behavior in some very specific circumstances on CentOS 7 systems. Namely, if the user already has repos installed that list a repository priority in their configuration (none of the official repos do this, including EPEL) and they do not have yum-plugin-priorities installed (which is exceedingly unlikely if they have repos with priorities configured), then installing Netdata may cause future system updates to install different packages than they would previously.
• May be seen as ‘underhanded’ by some users because it’s not clearly visible in most cases what was done.

Explicitly exclude netdata packages from consideration in EPEL

Pros

• Officially recommended solution for situations like this.
• Would be 100% transparent to users except for those who were affected by this issue.
• Avoids unexpected side-effects for most users.

Cons

• Not actually technically feasible:
  • Requires EPEL to be installed for it to be possible (and us installing EPEL only to not actually install any packages from it looks rather bad from a PR perspective).
  • Actually doing it in a reversible manner on a live system is exceedingly complicated.
  • Will stop working if EPEL is later uninstalled and reinstalled.
  • May cause updates to the EPEL repository configuration by the package manager to stop working.

[cpipilas]

From the product point of view, I would be in favour of the second option (Set a high repository priority for our package repositories):

• Transparent for users
• Controlled side-effects
• Enforcing ordered protection of repositories decreases potential reliability/maintainability issues

[iigorkarpov]

I disagree that using the priorities is either good or desirable. 7. A Cautionary Note [image: <!>] Note: The upstream maintainer of yum, Seth Vidal, had the following to say about 'yum priorities' in September 2009: Gosh, I hope people do not set up yum priorities. There are so many things aboutpriorities that make me cringe all over. It could just be that it reminds me ofapt 'pinning' and that makes me want to hurl. [image: <!>] This matter was discussed in more depth in the mailing list thread starting here <http://lists.centos.org/pipermail/centos/2009-November/086189.html>. The Repositories <http://wiki.centos.org/AdditionalResources/Repositories> article noted in that thread, which discusses the exclude and includepkg options for yum, is a better place to start in understanding priorities. The primary concern is that priorities is heavy handed over removing packages from the transaction set. It makes it difficult to readily determine what packages are being ignored and why. Even so, it is very flexible and can be extremely useful to provide the largest available list of packages.

…

On Mon, Mar 14, 2022 at 5:36 PM Costas Pipilas ***@***.***> wrote: From the product point of view, I would be in favour of the second option (Set a high repository priority for our package repositories): - Transparent for users - Controlled side-effects - Enforcing ordered protection of repositories decreases potential reliability/maintainability issues — Reply to this email directly, view it on GitHub <#12331 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOOJNNT4XUJ3CUDELNP7SLU75MJLANCNFSM5QDESFPQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Regards, -- Igor

[ilyam8]

@iigorkarpov that is a quote from 2009. Not really an argument.

• you strongly disagree with using priorities.
• you suggest using explicitly exclude Netdata package from consideration in EPEL - this won't do according to @Ferroin (see Cons).

So what do you think we need to do? If you think that excluding Netdata packages is a way to go, then what do we do with the Cons?

[Ferroin]

I’m not saying that priorities are a good or even desirable solution, I’m arguing that they are the only option that actually solves this issue for all affected users.

We CAN NOT persistently exclude the Netdata packages from EPEL, because doing so requires modifying the EPEL repository configuration on the system itself, which in turn requires us to install EPEL if it is not installed even if we do not otherwise depend on it.

We SHOULD NOT rename the the packages, because doing so does not actually solve anything for existing users, impacts all users negatively whether they are affected by this issue or not, adds a nontrivial maintenance burdern for us, and may eventually not actually solve the issue at all (because Fedora (the source for EPEL) and other distros may just change to match anyway).

[iigorkarpov]

Just a remark. We do install EPEL on RHEL 7 systems anyway, because the netdata package depends on the packages in it. Judy, libuv, etc. Even RHEL 8 has one package (and its dependencies) from EPEL. I opened another discussion about the possibility to replace netcat with nmap-ncat in order to get rid of EPEL on RHEL 8. (Here I assume that Rocky is almost identical to Centos/RHEL.) As for the rest, I suggest closing the discussion. I stand by my opinion, though.

…

On Wed, Mar 16, 2022 at 4:39 PM Austin S. Hemmelgarn < ***@***.***> wrote: I’m not saying that priorities are a good or even desirable solution, I’m arguing that they are the only option that actually solves this issue for all affected users. We *CAN NOT* persistently exclude the Netdata packages from EPEL, because doing so requires modifying the EPEL repository configuration on the system itself, which in turn requires us to install EPEL if it is not installed even if we do not otherwise depend on it. We *SHOULD NOT* rename the the packages, because doing so does not actually solve anything for existing users, impacts *all* users negatively whether they are affected by this issue or not, adds a nontrivial maintenance burdern for us, and may eventually not actually solve the issue at all (because Fedora (the source for EPEL) and other distros may just change to match anyway). — Reply to this email directly, view it on GitHub <#12331 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOOJNLGEFMX6AMJA2VTLSDVAHXBHANCNFSM5QDESFPQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Regards, -- Igor

[iigorkarpov]

I suggest exactly what I suggested - renaming the package. Sorry if I did not myself clear. As for the rebuke about the age of the view I cited about the undesirability of using priorities, I will say two things. 1) It was about Centos/RHEL7. It came out just in 2009. It has hardly changed enough since then for the statement to no longer be relevant. Otherwise this quote would have been removed from Centos wiki years ago. 2) It was said by the person who was developing yum at the time. I think I wouldn't be wrong in assuming that he had a pretty good grasp of the subject. The situation, as I wrote, is much more complicated than it seems. In addition to priorities, there are the 'costs' that I mentioned. Costs also affect the installer's decision about which package to choose. Consequently, the use of priorities does not provide a hundred percent guarantee, being a potential source of confusion. There is nothing wrong with using exclusive names. Many companies do this. For example, the ISC version of their BIND package is called isc-bind-bind. And there is also isc-bind-bind-utils. The same goes for DHCP. And last but not least. You can't treat the user's computer as if it were your own. Ideally, interference with the configuration should be minimal. The Netdata agent is just a monitoring utility. Don't act like it's the center of the world. It is not.

…

On Wed, Mar 16, 2022 at 3:17 PM Ilya Mashchenko ***@***.***> wrote: @iigorkarpov <https://github.com/iigorkarpov> that is a quote from 2009. Not really an argument. - you strongly disagree with using priorities. - you suggest using explicitly exclude Netdata package from consideration in EPEL - this won't do according to @Ferroin <https://github.com/Ferroin> (see Cons). So what do you think we need to do? If you think that excluding Netdata packages is a way to go, then what do we do with the Cons? — Reply to this email directly, view it on GitHub <#12331 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOOJNNDIOTJWFFXDTJDUBLVAHNPVANCNFSM5QDESFPQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Regards, -- Igor