Input Lag Compensation to compensate for game's internal lag?

RetroArch General
#61

Thanks.

There’s an issue when using shaders with the unlag edition. When using D3D11 part of the shader will be show upside down. See attached image of Addams Family in Snes9X with D3D11 and crt-easymode-halation.slang shader. It differs per shader, some will just show the entire image upside down and some will only show some parts upside down and the rest normal.

With GL there is no immediate visible issue when using a shader, but upon closer inspection some passes don’t seem to get applied. Try for example crt-easymode-halation and raise halation value to some large value. You’ll see that nothing happens when using the unlag edition.

addamsfamily_RA_unlag_d3d11_crt-easymode-halation
addamsfamily_RA_unlag_d3d11_crt-easymode-halation.png1280×720 491 KB

#62

How about Internal Latency Compensation? It’s a bit cumbersome and not very snazzy, but it’s at least somewhat descriptive. I’d like to have the word “internal” (or something similar) included, since that indicates the compensation is made for latency that is internal to the game (or core).

I guess “Lag Frame Removal” or “Lag Frame Reduction” could be used as well. Just slightly more verbose than your Lag Reduction proposal. Still, the word “lag” can mean different things to different people, so “Internal Latency Compensation” might still be preferable.

2 Likes
#63

I like “Internal Latency Compensation”. :slight_smile:

1 Like
#64

Are we sure yet that this internal lag frame reduction method is as robust as frame_delay has proven to be? Where frame delay will leave original gameplay untouched, I’'m not (yet) sure whether this feature is the same.

What happens when game logic and A.I. (enemy placement) responds to player input with a different lag than the main player sprite? (For example maybe in fighting games to tweak the difficulty?)

The below is just something made up to show as a possible edge case.

Case 1:

Player sprite [k]=3
Enemy sprites [k]=3

Internal lag reduction of 2 will show next frame response for both.

Case 2:

Player sprite [k]=3
Enemy sprites [k]=1

Internal lag reduction of 2 will show next frame response for the main Player sprite, but what is happening to the Enemy sprites? Does it have a “lag” of “-1”, or is it skipping a frame in the output that the player should be seeing normally?

Note, I love the developments in this thread :grinning:

#65

It will skip a frame that the player should normally be seeing. While the feature itself should be completely robust as long as state can be saved and restored seemlessly, the applicability of the feature really depends, as you point out, on if the games consistently respond in a certain amount of frames. Determining if you’re “losing” a frame here or there might be hard in practice and thankfully the experience degrades gracefully, i.e. there’s no actual decrease in overall framerate. This is important to stress: In case the game responds faster than what this new feature is configured for, you’re not losing a frame in the traditional sense, but reactions to input would instead become visible suddenly instead of smoothly.

#66

Here’s an example, let’s say Super Mario Bros.

If you were to run ahead 2 frames, you would see Mario jump at a higher initial position than otherwise; it would have skipped one frame of his jump animation.

The game only has 1 frame of internal lag, and exceeding that would mess with how the game looks.

#67

@Brunnis

I agree that a descriptive name is better. I don’t like how a lot of projects are starting to pick cool names that don’t tell people shit about what it actually does. While we have primarily talked about it compensating for internal latency, it doesn’t really have anything to do with that in and of itself, and indeed using greater numbers of frames will quickly exceed internal latency in many cases. Therefore, I would lobby for a name that includes terms like “lookahead” and/or “rollback”.

2 Likes
#68

while i’ve always referred to this as a “leapfrog frame stepping method” (lifting the term from the similar(ish) numerical integration technique) in discussions about emulation latency, i knew it wouldn’t be an appropriate name to expose to the end user. up until now i’ve always thought “latency compensation” would be the most appropriate term, especially since gens-rr has apparently had the feature since at least 2013 and that’s what they refer to it as, but i really like the idea of something with “lookahead” in the name. that’s definitely more descriptive.

below is some info i wrote up on the feature that might be appropriate for use in the frontend. IDK, maybe it can be used as a starting off point? i consider my writing and language skills to be sub-par, so hopefully whomever is in charge of it can clean up whatever they may choose to lift from it.

internal name: lookahead_frames

exposed name: Lookahead Frames (Unsafe and Experimental)

description:

Can be used to compensate for a games internal latency by stepping ahead N frames in order to output video and audio from an extrapolated point in the future, effectively removing N frames of input latency. This feature requires that a core support serialization. Core serialization must also be complete and deterministic, which is difficult to guarantee. If this condition is not met by the core unexpected behavior and corruption may occur. Setting this too high may result in visual and audio glitches from an incorrectly extrapolated state, thus experimentation must be done on a content by content basis using trial and error and/or single frame stepping. However, a setting of 1 should be appropriate for nearly all supported content.

DO NOT use this setting unless you understand what it does and know what you’re doing. DO NOT report bugs that occur while this feature is enabled, unless the bug is specifically about the behavior of said feature.

1 Like
#69

Telling people not to report bugs may result in things like the GB Skate or Die stage 3 issue getting missed.

As for terminology, there is also things along the line of “Show future frames”. I’d probably call it “Game internal lag compensation”.

Then there’s audio, and deciding which audio frame to present to the player. We want to avoid missing the first 1/60s of sound effects triggered by a player’s action. It just so happens that taking the audio frame from the displayed frame works well most of the time. We don’t need anything extreme countermeasures such as crossfading, since the audio works so well already. However, a few games do play the sound effect immediately, such as 3D World Runner. Or some games might lag for 2 frames, but play sounds after 1 frame.

As for core compatibility, I’d use testing, then whitelists of which cores are allowed to use the feature without disabling the whitelist.

#70

hence why i added “unless the bug is specifically about the behavior of said feature”, which in this case it was, no?

IDK, i think you may be over thinking it. this is a hack, a extremely desirable and useful one but a hack nonetheless. it’s never going to offer “perfect” results in all situations no matter what you do. personally, i think it’s a feature that’s appropriate for power users only and should be hidden by default via the menu_show_advanced_settings. but my opinion is irrelevant, you’re doing the grunt work and you and the maintainers will ultimately decide where and how it’s exposed.

please don’t take this comment as being negative, i’m simply trying to be objective. let me take this moment to express my gratitude to you and the maintainers for your work on getting this feature implemented and the various cores up to par.

#71

Could the RA GUI pull this information from the core’s .info file?

If so, perhaps a new attribute could be added to the info file along the lines of lookahead_supported = "true"

Example .info file: https://github.com/libretro/libretro-super/blob/master/dist/info/snes9x_libretro.info

#72

For 8-bit and 16-bit games, if the feature works well and stable, you’d just want to simply leave it on all the time. Then you’d look up the game, and how many frames of lag it has, and use that, otherwise assume 1 frame of lag.

Pretty much every game will have at least 1 frame of lag, except for Atari 2600 games. (Yes, I’m aware of menu systems fast enough to run during vblank time, those also have no lag)

#73

In RA you can always do overrides for core/games.
Activating only for what works for you.

#74

We have some design philosophies that might cause some issues with the absolute safest, most user-friendly menu implementation:

1.) RetroArch isn’t supposed to know anything about specific cores beforehand. That is, it can’t have an internal/external database of cores that are compatible with a given feature because anyone should be able to write a libretro core and expect it to work without us “approving” it directly or indirectly.

2.) Core info files shouldn’t include information that only matters to RetroArch (i.e., and not other frontends). However, we could probably come up with some useful, frontend-agnostic information nugget, like “serialization support” or whatever, since that’s valuable information for any frontend that wants to monkey with time. I don’t think there are currently any options that are forcibly disabled and/or hidden from users based on core info files, but I’m not against the concept.

#75

Does Retroarch have any standard means of telling a core that it is okay to discard a video frame, or discard an audio frame?

edit: Looks like it does NOT have a standard way. The Snes9x core does have code that handles skipping frames, so if I could use a standard way of telling the core that the frame it’s working on will be skipped, that would be helpful.

Anyway, Snes9x 2010… Got the sound almost working but not quite. Mostly smooth but it crackles periodically. I hate those kind of issues.

Edit: Got it working! Check the core updater in a few hours.

2 Likes
#76

Yes, “look-ahead,” finally a sane and simple name. (Nerds trying to name something technical = entertaining! :smiling_imp:)

One (objective) comment: This is output lag compensation, not input lag. Input lag is anything delaying delivery of my input to the game.

Input latency, output lag.

#77

Hi

Is there a special way to activate this feature or do I have to download the test build in post #2 of this thread?

#78

That’s a good idea. I like the “Lookahead Frames” name that @e-tank came up with.

That’s a shame. That means you’ll have to experiment and probably apply some game specific overrides.

You still have to download the test build.

There is no set definition of what “input lag” means, but from what I’ve seen, the more common way to use the term is to describe the overall effect, i.e. “There is lag between applying input and receiving a response”.

#79

Just made a quick test with Snes9x2010 and audio seems to work fine. Great job, @Dwedit!

I did observe a curious behavior on Snes9x that differs from Snes9x2010 (and QuickNES): When you return to the Quick Menu or pause the emulator with ‘p’, the screen goes black. Why is only Snes9x doing that?

#80

Goes black screen on pause with GPGX too.

Did I understand correctly that there will be a problem with games that have only 1 frame of internal lag?
Also, will Nestopia be fixed?