Shumway fork - Pixi-SWF

[ivanpopelyshev]

https://github.com/pixijs/pixi-swf

I forked Mozilla Shumway. I've been working on this for 7 months already.

Goals

• Help developers migrate their old ActionScript projects to PixiJS and TypeScript
• Integration with tools that export SWF format: vector graphics and UI for PixiJS applications

One of short-term goals is to make a sandbox for people to try our heavily modified shumway API and a number of examples.

Project does not need big advertising yet, only contributors. It would be possible to port on Kotlin/Swift/whatever-you-want later, first we need the best coverage of flash api`.

Have a cat burger: http://pixijs.io/pixi-swf/demos/ninja-cat.html

Source: http://pixijs.io/pixi-swf/demos/ninja-cat.js

[Brian151]

Might I ask you something?

I see that in your comparison, the original Shumway render has an issue which no tool to date (Flash included) has been able to correct when shapes are converted to other formats. Your Pixi-SWF thing seems to lack this. How did you get the fills to not have those weird and extremely ugly/distracting gaps between things?

SVG exports have it
JPEXS FFDEC has it
HTML5 animations exported via Animate have them
A tool I was writing has had them.

I'd like to know, because one of the things that I thus far hate about converting flash assets is that Vector art has a tendency to get ruined unless it's actually rendered within flash.

[demurgos]

Ivan calls it shape seams, and it is also one of the long-standing issues I had with most decompilers.
This is caused by the renderer. This is not a shape decoding issue. SVG and canvas shapes are rasterized sequentially with anti-aliasing. WebGL, custom blending or shape modifications fix it.

Here is a minimal example reproducing it:

• Canvas: https://jsfiddle.net/9zs0pdc9/3/
• SVG: https://jsfiddle.net/9zs0pdc9/23/
• Edit: SVG with crisp edges: https://jsfiddle.net/Lbzm1vj9/2/

This draws two white triangles that should form a square, but you can see the diagonal:

On the diagonal, the pixels of the lower triangle cover half of the pixel so they get the color "white with 50% opacity". The pixels on the diagonal of the upper triangle have the same color. Now the triangles overlap on the diagonal: their colors are combined in the final image. Normal blending with rgba(255, 255, 255, 0.5) on top of rgba(255, 255, 255, 0.5) returns rgba(255, 255, 255, 0.75). This is white with 75% opacity: because it's not 100% you see the red background.

The solution is to either use a different blending equation or draw the shapes in deferred mode instead of anti-aliasing immediately. PixiJS uses a new blending equation. I believe that it draws the shapes on an off-canvas layer, fixes the alpha value and then draws the result on the main canvas. There are some edge-cases with this solution but it works pretty well for most of the shapes. @ivanpopelyshev could maybe provide the exact details of the blending and how it works.

I think that Adobe's renderer uses deferred rendering instead. You can achieve it with WebGL but it's more complicated for 2D shapes. You first need to tessellate your shapes into triangles and then deal with WebGL. PixiJS already has the mechanisms there (using earcut for tessellation) but I understood that it's a lot of work to move Shumway to WebGL.

Example using WebGL: https://jsfiddle.net/9zs0pdc9/49/

[ivanpopelyshev]

Right now its not even using PixiJS, I patched many things in original Shumway renderer, and even when I move it to Pixi, parts of that renderer will continue to exist in CanvasRenderer fallback.

My idea is that paths that have those problems are on the bottom and do not intersect with each other, they share some edges. When the shape is prepared, I mark those paths, so when renderer is drawing it into separate layer - it can use Additive blendmode instead of Normal.

I know its not a full solution, its just a heuristic that needs extra tweaking. For now its very stupid, but the more cases i'll get from users, the better it will become. The game im working on has 500 megapixels of animated content, I believe that's enough for me to make this thing smart.

Here it is:

1. rendering https://github.com/pixijs/pixi-swf/commit/d9c7fbf831aa47eb5d5a3f1558bfeae3ad07d4d0#diff-b2011b0c661dee4fac28277c1c238bccR740
2. detection https://github.com/pixijs/pixi-swf/commit/d9c7fbf831aa47eb5d5a3f1558bfeae3ad07d4d0#diff-b2011b0c661dee4fac28277c1c238bccR678

For WebGL it will be difficult, because we dont have AA in framebuffers there :(

SUMMARY: lucky shumway architecture + my lucky guess :)

[Brian151]

Well, lotsa fancy terms thrown-around there!

I see what you're saying, the alpha thing makes sense, and I actually tried something to correct that before...

I'll look into this more deeply later, kinda keep having...stuff........ coming up... (and it's really starting to irritate me)

[demurgos]

Regarding SVG, I found this SO question. I changed my SVG example to use it:

<svg style="background: red;" width="100" height="100" version="1.1" viewBox="0 0 100 100" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <filter id="null">
      <feBlend mode="normal" in="SourceGraphic"/>
    </filter>
  </defs>
  <g filter="url(#null)">
    <path d="m20 20v60h60z" fill="#ffff"/>
    <path d="m20 20h60v60z" fill="#ffff"/>
  </g>
</svg>

It's better but still not ideal. The gap is less visible but you get get blury edges:

[ivanpopelyshev]

I've deduced that thing three years ago. Its worse than additive blending but it has no visible gaps, it filters all the "bad" alpha >0.5

  <defs>
  <filter id="filter">
    <feComponentTransfer color-interpolation-filters="sRGB">
       <feFuncA type="table" tableValues="0 1 1"></feFuncA>
    </feComponentTransfer>
  </filter>
  </defs>
  <g filter="url(#filter)">

[Cleod9]

Love the ninja cat demo! Is it just me though or is performance in Chrome significantly better than Firefox? I've been having this issue often with HTML5 canvas and I've never been able to confirm if it's the browser or the canvas implementation that is the offender

[ivanpopelyshev]

@Cleod9 You cal also zoom in/out in browser and refresh the page, that'll give you better image but hurt performance.
For chrome it slightly lags on 2560x1440 my old video (GT 620)
For firefox i dont zoom it and i see lags.
I hope it'll be much faster when we move it to WebGL.

[Cleod9]

Thanks @ivanpopelyshev , that's pretty cool that the vector art is scaled based on the window zoom! I suppose it wouldn't be possible to recalculate that at runtime due to how expensive it would be to re-rasterize everything... Anyway, be sure to update us once WebGL is up and running!

[ivanpopelyshev]

Its possible, I just didnt implement canvas resize for the demo. And its rarely when engines do change resolution (x1, x2, x4) in runtime.

[Brian151]

For...other reasons, I have looked-into the dynamic re-sizing of an HTML5 canvas.

It's totally possible, but not necessarily fun/easy to do. You actually have to re-establish the context because the current one will be broken if width or height are altered (which tbh, i think was a very stupid way to implement this, but maybe someone knows something i don't that suggests otherwise?) At least from when/where I last looked, finding much information about it at all is extremely difficult. And the explanations tend to be less than clear. It's a mess... Probably why HTML5 games don't seem to do it. (even where it'd be really useful/neat)

Indeed it's something that'd be done rarely, or ideally, anyways. I sometimes fine-tune the sizes of program windows for very specific situations, and in those cases, they're getting resized a bit more than normal. Still not overly excessive, ofc...

[ivanpopelyshev]

Resize is not a problem, at least in pixi applications, the resolution change is, you have to download adjusted resources or to redo the vector, but it also is not a big problem.

The problem is to make everything together so people just use this thing like Flash.

[Brian151]

At least when dealing with vectors, re-sizing shouldn't be a terribly difficult problem. Bitmaps/Rasters, however... I didn't realize Pixi could do this by itself. I only recently learned of it, and seeing too many games/engines stumble with being stable, or displaying enough graphics [properly] to be playable, I've tended to avoid it. This has been especially true if they try to use webGL... I'm not sure if its those developers/engines are Pixi, itself, but it falls in a category of HML5/webGL games that for me have a majority percent of MAJOR graphics errors and complete crashes. I might start looking into ways to force regular context rendering on these games and see if that fixes some of the issues I've had.

a little lost on what you mean by the second statement...

[ivanpopelyshev]

I'm familiar with huge amount of problems that people have when they use HTML5 graphics.

There are people who coded more lines than me. There are many people who have a piece of puzzle to make a successor for Flash, but no one with full solution. I want to gather all the data and actually make it with pixi-swf. Pixi is just an architecture, the correct name would be pixi-three-shumway-openfl-libgdx-swf but its too long :)

[Brian151]

I have a fine mess of a game library and most of the effort spent has been graphics stuff.
https://github.com/Brian151/UnNamed-html5-Game-Library
Also likely to re-code this thing AGAIN... or maybe scrap/replace it

You're right, there... The same happened with my "Earthquake" project, an attempt at decompiling Director/Shockwave... We've pulled so many resources from other places/people... Still do, but there are plenty of contributions we've now made, ourselves (or at least, actually documented somewhat decently) I have intents of forking certain parts of SWF (or concepts, at least), and I definitely want to play with post-XFL FLA. https://github.com/Brian151/Various-Projects/tree/master/Flash/fla_buster
I kinda took a break though because of how messy code was getting, "shape seams", and trying to wrap my head around doing so much as to re-construct the full edges since flash optimizes them. Also ran-into problems with having redundant move() commands/records. One of the things you'll need eventually is going to be a 1:1 copy of the ActionScript API, right? (and Stupid question, but do you have the SWF specs? I finally just downloaded SWF and AMF specs cuz they kept moving/disappearing a lot. )

Agreed the name would be too long. XD

[ivanpopelyshev]

Thank you for the links!

redundant move() commands/records

Working on it. I'm gonna patch the chromium.
https://bugs.chromium.org/p/chromium/issues/detail?id=824145

[Brian151]

iirc, i simply just removed them since...really, they aren't necessary. (at least, don't think they are, right?)

[ivanpopelyshev]

Yep, I removed them too, as a workaround.

[Brian151]

kinda funny why a format focused so intensely on being small would do that...
(not that XFL apply...XD)

[demurgos]

and Stupid question, but do you have the SWF specs?

@Brian151
I keep all the relevant specs I find there: https://github.com/open-flash/open-flash/tree/master/specs

[Brian151]

i did not realize those were there. iirc, my earthquake organization has chosen not to share "official" documents cuz of copyrights or something??? ill check on that again... kinda figured everyone else would do the same. Silly, considering the documents are completely free and all...

[rob-bateman]

@ivanpopelyshev really interested in what you're doing here, whats the best way to get in touch with you for projects?

[ivanpopelyshev]

@rob-bateman :)

[cmann1]

Firstly, thanks for this. I've also been trying to extract vector sprites from fla files and encountered this problem with shape seams, and this was the only place I could find that really discussed it in more detail.

So the additive blend mode works well and with my initial tests seemed to solve the problem, but later on with other sprites the results weren't as good.
I might have another more full proof solution, though it might be slower as it probably requires some extra steps. I do have it working (so far) and 'll post an overview of how it works below, but I'm a little busy right now so I'm not sure when I'll be able to get round to cleaning up the code and making a workable demo I can post. So for now I' just wanted to put this up, sort of as a reminder to myself.
Also the sprites I'm working with are just solid filled shapes so I might be overlooking somethings, filters, strokes, groups, etc. I'm not sure.
Plus I might be making some assumptions that are actually incorrect, but so far it seems to work well for my sprites.

So basically what I propose is:

1. Break all shapes into groups when extracting from fla/swf. By this I mean any set of shapes that don't overlap and might share edges ie. layers and groups.
2. Next, for each group, merge all shapes to create an outline/silhouette. Basically find and discard shared edges which is fairly trivial, then recombine the remaining edges into a single path. This is the most complex part and might fail with certain paths but is working for what I have.

Now when rendering a group:

1. Render each path. Possibly with the additive/lighter blend mode for the best shape seams.
2. Render it back onto itself with a 1px blur. This should fill any gaps between shapes, but erything will be blurred (obviously) and there'll be a halo around the shape.
3. Again render the paths normally on top of everything. Normal blending and no filters.
4. Finally mask the results of the previous three steps using "destination-in" and the group's silhouette created previously.

[ivanpopelyshev]

Yeah, your ideas are good.

My current implementation uses webgl, and when i upload something, I change pixels alpha:

		fixImageData(imageData: ImageData) {
			const data = imageData.data;

			let opaque = 0;
			let transparent = 0;
			for (let i = 3; i < data.length; i += 4) {
				if (data[i] > 0) {
					if (data[i] == 255) {
						opaque++;
					} else {
						transparent++;
					}
				}
			}

			if (opaque * 99 > transparent) {
				for (let i = 3; i < data.length; i += 4) {
					if (data[i] > 153) {
						data[i] = data[i] * 10 - 153 * 9;
					}
				}
			}
			return imageData;
		}

		onTextureUpload(renderer: gobi.pixi.WebGLRenderer, baseTexture: gobi.core.BaseTexture, glTexture: gobi.glCore.GLTexture) {
			const gl = renderer.gl;
			if (glTexture._updateID < 0) {
				gl.texImage2D(
					gl.TEXTURE_2D,
					0,
					gl.RGBA,
					gl.RGBA,
					gl.UNSIGNED_BYTE,
					this.fixImageData(this.context.getImageData(0, 0, baseTexture.width, baseTexture.height))
				);
				return true;
			}

Of course I detect which shapes are better to deal on canvas 2d and which ones are simple enough for pixi-v5 Graphics algorithm. If shape has transparent pixels, i dont change alpha.

I didnt publish it yet, too much time working on pixijs-v5. This thing shows very heavy app with hundreds of SWF loaded in runtime and TONS OF FILTERS at 60FPS. I even implemented all Blur-based filters, they are very close to flash.

[Brian151]

@cmann1
FLAs, hm?
XFL, i assume?
(seeing how there's insufficient information on the 'binary blob' format)

For what reason are you messing in FLA?
What is it that you hope to gain/achieve/build?

I also had started working on a tool, but between shape seams and the 'joys' of XML-parsing...
(and a handful of other things)
It's kinda sitting untouched currently...

Something a friend of mine observed is 'unedited' shapes in his FLAs are actually changing if they're open in the IDE (like if they're on the main timeline, most note-ably) I didn't yet look into it, but I have a theory.
The flash IDE ALWAYS adds those extra 'move' commands (!)
Maybe each path is actually meant as a series of segments that can be re-ordered at whim?
(since the move at the beginning of a segment matches the target of a line or curve of another)
consider also how the fillstyles work. each edge record can have 2 fill styles. and to achieve this, it means for one of those fill styles, the edge segments have to be reversed in order)

My actual theory for this happening is more how flash processes/renders the shapes, in general, though. It can display some pretty deeply nested display trees/lists, but how? consider why GPUs are being used (perhaps over-hyped, IMO...) to make rendering faster. The idea is that the GPU can parallel-process whatever graphics need to be drawn, vs the in-order, one-by-one, and possibly recursive implementation that'd be seen in software. The CPU technically can, but it was never designed with this as its main goal. The CPU is meant to process long sequential lists of operations with ideally minimal/no multi-threading. Is it possible flash maybe multi-threads its code for this? Either way, we know flash runs on the CPU... I will say, when you consider how deeply nested movieclips can become, with effects applied, transforms, etc, the performance flash is capable of achieving is somewhat shocking. Even for native code, that's a LOT of work for the CPU to be doing. (and then there's also the AVM to execute the AS bytecode!) Consider also a starling and/or dragonbones animation. These perform well when flash player can use stage3D to leverage the GPU. But, if they are run in software mode, the performance is arguably WORSE than if the original native flash movieclips were used. And this is likely a combination of sequential processing, recursive function calls, and doing it all in a high-level language. (well, and maybe an abstraction or two added by the starling/dragonbones frameworks, themselves)

@ivanpopelyshev
interesting...

Something you should consider with webGL, though, It's simply not a magic bullet it claims to be. Not carefully considering the capabilities of the system or browser will haunt you (or at least, torment your users) I wouldn't recommend something that is so dependent on GL that it won't work well (or even crashes) without... I've seen MULTIPLE games fail to keep this in mind. The result? I can't play them, or MAJOR graphics errors that also render them unplayable due to important text and graphics not being displayed. Some also crash because the errors propagate straight into the main game code and aren't handled correctly. The same is true for websites. They're so rushed to use bleeding edge technology that they've alienated a certain portion of their users.

[cmann1]

@ivanpopelyshev Interesting. Care to explain? I see what you're going for, but what specifically is this opaque * 99 > transparent and this data[i] = data[i] * 10 - 153 * 9; doing?

@Brian151

XFL, i assume?

Actually no. That's an xml format newer flash versions use, right? I had forgotten about that but after looking the FLAs I have aren't anyway.
I used a script to export each frame as an FXG which, at least for my usage, seems pretty simple and can be converted to svg.

For what reason are you messing in FLA?

Because that's the only format I have them in.

What is it that you hope to gain/achieve/build?

It's a small side project - I'm working on a tool to recolour sprites for a small indie game. The devs were nice enough to let me have the source files for the artwork, which are all FLA.
So it's got nothing really to do with pixi, but I thought it might be worth adding to the discussion.

[Brian151]

@cmann1
I'm more interested in handling the XML format directly. (and probably SWF, too)
I did briefly mess with FXG for one of my projects, and it wasn't really cutting it
Ofc, I didn't have any scripts, but still...

Yes, XFL the newer [MOSTLY] xml format.

May or may not deal with FXG, at least it's truly an open format
I contemplate not even getting INVOLVED with the binary blob FLA
That'd be a lot of effort for arguably not much gain.
But then again, my earthquake project has the stated goal of breaking shockwave/director wide open, so maybe in the future???

In all honesty, I'm not terribly interested in SVG conversion

Your reasons make as much sense as something I would come up with, so ok

I also do not plan to achieve my goals through Pixi. If I do rendering, it'll be through my own game library. One big challenge I see that even createJS has failed to address, is certain image filters/effects. They can't currently be mirrored reliably and are just too expensive to be practical. (at least in JS... which forces me to accept performance hits, or go down the webGL rabbit hole)

My goal actually rather is to provide an abstraction layer by which to read data from a SWF/FLA, and leave it to other code to do anything meaningful. Taking this approach with a lot of my libraries since it in theory maximizes customization options. the library provides a mean to read the file, and might do some slight conversion. However, it won't add the overhead and possibly implementations of forcing a default rendering/playback implementation. We'll just see... I may or not provide a default implementation, it just won't be so tightly coupled to the library that you practically need to use it.

I first need a better way to address XML-parsing...