I will briefly summarize Shortcode API changes since WordPress 4.0 and then kick off some ideas for a roadmap.
The first major accomplishment was the expansion of the API documentation, including a new large section I wrote about the formal syntax for shortcode input.
I also put forward a robust parser concept for the function wptexturize() that promised to re-introduce the ability to use unrestricted HTML code inside of shortcodes and shortcode attributes. That concept went through many, many changes before being introduced in v4.2.3. After consulting with the WordPress security team, and after extensive testing of the shortcode parsing functions, we determined that the shortcodes-first parsing strategy was fundamentally flawed and could not be included with any version beyond v4.2.2. This is why I added an HTML parser to the Shortcode API and ultimately curtailed the use of shortcodes inside HTML rather than expand the use of HTML inside shortcodes.
