I think we can do this for the next major release of ggplot2 (i.e. this can be a project for my next summer intern). The basic implementation is much simpler than I'd feared; the work will mostly be carefully considering which theme properties we need and then performing a routine transformation for each geom. (We'll probably also need to use a SE variant of aes() since we I think don't want any delayed evaluation in the defaults)

Sounds great!

As themes grow in scope, I wonder if one could also consider developing an alternative syntax for them, based on css. (I'm thinking of the way d3.js often uses css to set default properties, although that's a bit inconsistent because of the css vs svg attributes mess).
It could start as a simple mechanism to parse a css-like description and turn it into a well-formed ggplot2 theme object, but perhaps this would evolve much further and trigger new ideas, especially for web-based output. One such idea is the beamer concept of themes, bg, fg, etc. Wouldn't it be neat if one could specify common attributes in a standardised format, and have plots, slides, etc. all use this consistently?

I was thinking about the same issue the last few days, but I'd like to propose a different angle of attack: Is there any reason why the theme can't simply store a list of default scales? Something like default_scales = list("scale_color_discrete" = scale_color_hue, "scale_color_continuous" = scale_color_gradient, ...)? The implementation should be pretty straightforward, and it wouldn't require a separate language to describe scale properties inside themes.

@clauswilke this seems orthogonal to the original issue. I agree that it would be nice to have default colour palettes associated with the theme (ggthemes comes to mind), but in my view (as I've argued before) the issue is that scale_colour_zzz have adopted a strange interface: rather than having just a few scales (discrete, continuous, gradient) and then selecting the palette from this common interface (palette = brewer:Set1), the current mechanism leads to a jungle of names (dichromat, brewer, viridis, tableau, fivethirtyeight, etc.) where all that's changing is a set of colours.
It would be much easier to specify a couple of default palettes in the theme, rather than a set of default scales. Besides the technical aspect, it also doesn't feel right with the grammar to put scales in the theme (why not geoms, and coords, and facets? – that would be unwieldy, and a serious technical challenge with ggproto etc.).

Just to clarify on this one issue:

it also doesn't feel right with the grammar to put scales in the theme (why not geoms, and coords, and facets? – that would be unwieldy, and a serious technical challenge with ggproto etc.)

I'm suggesting to register the default scales with the theme, not make the theme the preferred way of changing scales. My proposal would change virtually nothing for existing ggplot2 code, other than that theme_gray() would by default use different point, line, and fill colors than theme_dark(). There are no default geoms, so they wouldn't go into the theme. There are default coords and facets, so one could make an argument for registering them with the theme, e.g. for a theme for map plotting. But it's much less urgent than scales, I think.

I understood your proposal but I still think it'd be awkward and foreign to the ggplot2 framework to start putting things like default scales in the theme. Note that geoms have defaults too: point shape, colour (and also stat, position, etc.). This proposal was to set the default visual aspects of geoms (unset and unmapped aesthetics), which is independent of the grammar. Scales, in their function, are part of the grammar, but a specific colour palette (within a class – continuous/discrete/gradient) should not have to be.

@clauswilke would you mind moving the discussion of associating default scales with a theme to another issue? I think it might be a good idea, but it is largely orthogonal to this issue.

@dpseidel I think the place to start is to figure out what are the primary geom defaults that need to go in the theme - i.e. points and lines have different sizes, but are there more variations? It would be useful to have a table with aesthetics in the columns, geoms in the rows, and the default values in the cells. You should be able to automate most of it starting from:

geom_names <- apropos("^Geom", ignore.case = FALSE)
geoms <- mget(geom_names, env = asNamespace("ggplot2"))
lapply(geoms, function(x) x$default_aes)

a while back it looked something like this – presumably somewhat different now

I've filed a new issue specifically about default scales in themes (#2691).

So I have pulled together a list of the current defaults and made my first attempt at an updated hack-- mostly for the benefit of my own understanding of the underpinnings of ggplot's internals.

By adding a new theme element geom.colour and passing theme to ggplot_build(), the theme element can be called in use_defaults() to override the colour default only when applicable for the geom and not already set by params or mapped to data.

After looking over the table of defaults and sorting it some into groups of similarities, it's clear that first and foremost the important defaults to be able to tie to theme are colour and fill but any framework I build would hopefully be extensible to other properties. I can imagine font family (easily tied to the overall text element in theme presumably) and perhaps alpha and size might be of interest. My first attempt addresses only colour and is not likely the most extensible approach to this problem: I am very open to advice and direction on better ways to proceed.

From a high-level perspective, my personal inclination would be for a system that uses inheritance to define harmonious and consistent defaults within a theme that "scale" most elements. In the spirit of rel() for sizes, which ensures that theme_bw(24) doesn't leave behind e.g a hard-coded 12pt title, I think it would be great to think of a similar strategy for colours, and possibly other aspects.

What I have in mind is a "foreground" (fg) and a "background" (bg) top-level setting, like Beamer themes, and specific theme children would build from those for their own shade, e.g. title.text would use fg as colour (typically 'black'), panel.background might use bg (e.g. grey98), but intermediates such as grid.lines might use a shade such as interpolate(fg, bg, 0.5) (implemented using e.g. colorRamp, or adjustcolor, or functions from Munsell such as mute(), etc.).

I would also re-advocate for consideration of a CSS parser + interpreter to define themes going forward (perhaps in the long-term). It'd be quite nice to use this extensible and widespread format, well-suited for the description of cascading styles. Since more and more R graphics target web-based output (plotly, svg, d3, vega, etc) this would be a natural way to specify fonts, colours, margins, even layout maybe one day. Anyway, this is kind of tangential to the discussion, but I see that Claus came up with the same idea recently re text limitations in grid.

PS: I wrote this with the usual theme defaults in mind, but I think it applies to geom defaults as well, for a consistent look (the archetypical example being a black-background theme: one would want grid lines, text, but also point outlines to default to white, otherwise they become invisible).

I'm all for going CSS. I think it's the correct long-term vision. The one issue that I ran into and couldn't immediately resolve a few months ago is that there doesn't seem to be a decent CSS parser available for R, comparable to, for example, xml2 for parsing html.

I wrote some regular expressions for simplistic CSS parsing, but it doesn't do the inheritance part of CSS, it just parses key-value pairs.
https://github.com/clauswilke/gridtext/blob/master/R/parse-css.R

I think css is way way way too complicated. Maybe a css like DSL would be ok, but it would be a huge amount of work to develop the parser and specification. I don’t think using css is worth the implementation cost, especially given that the current theme system isn’t that bad.

The goal here is to find a simple way of controlling geom defaults from the theme, not a complete overhaul of the theme system.

I also failed to find one, but it's probably a case where there's little sense in reinventing the wheel: I don't know this area but R seems to have good support for seamless embedded javascript/html engines (V8?), so I imagine leveraging their functionality by interfacing with R would be the most robust and straight-forward approach. Maybe have the tool produce a json AST, then converted to a data.frame or nested list.
Only a specific subset of the full CSS specs would make sense, so perhaps a specialised validator would also be required, together with a comprehensive doc.

@hadley I think the two aren't mutually-exclusive: the CSS subset idea is more of a long-term, alternative spec, that could be parsed as an AST and then converted into a ggplot2 theme object.

I agree that right now it would seem a serious overkill, but going forward I imagine grid will need a successor, and IMO it would make sense to start thinking of a broader context for R graphics, with modern accepted technologies and formats (ie thinking in terms of, and leveraging, powerful tooks like css and svg attributes).

@hadley Yes, exactly what @baptiste is saying. @dpseidel should continue doing exactly what she is doing. We're just trying to formulate a vision of where things could go in the future. Maybe off-topic, but also broadly related.

I arrived at HTML/CSS from the perspective of wanting to add text labels that can change formatting half-way through, e.g. with one word in italics or in a different color. We could invent a new R DSL for that, just like plotmath, but I think going with existing standards is better. And also, I think there is a smooth transition from the current theme system to HTML/CSS that keeps the current theme system relevant all the way to the end:

1. First add a text grob that can render HTML. Styling can be injected via inline CSS. I have a proof-of-concept for that already. It could be integrated into the current system via element_text_html() or similar.

2. Then provide the option for a standalone CSS style sheet for a ggplot2 plot. Initially, only the html text grobs would use those settings, all others would ignore it.

3. If 1 and 2 works well, more rendering could be shifted over to html, and the theme system could generate CSS code that is injected into the style sheet.

4. In the end (and we may never get there), internally everything is HTML/SVG/CSS based. At that point, the theme system just provides a convenient way to generate the CSS. Nobody wants to write CSS by hand anyways.

I think moving away from DSLs implemented in R (internal dsls) to DSLs implemented in another language (external dsls) is a bad idea, especially given the poor match between the very general needs of HTML and CSS, and the specific needs of a visualisation system. It feels very "untidy" to me.

... That last comment doesn't seem well reasoned to me. HTML and CSS are literally for purposes of visualisation. You could argue that they have wider scope for other reasons, but just being a visualisation system doesn't necessarily imply more specific needs than another visualisation system. It's a bit of a nitpick though, as I agree that adding another layer of syntax is not probably not helpful. I mean... if that route were followed through with completely, then why not write a d3 generator instead of using ggplot2 in the first place?