Reintegrate projectM Visualizer

Gustavo L Conte

Here's a very simple milkdrop preset to test and compare running it inside Strawberry ( libprojectM v3.12 + Qt6 ) against the projectMSDL official visualizer. The results are pretty disappointing, since on the SDL app, it renders as expected, while on Strawberry the screen just flashes (at least the right color)

[settings]
presetAuthor=Gustavo L  Conte
presetName=000 Simplest Waveform for Strawberry Music Player debug

[init]
# Initialization code here
nWaveMode=2  # Use the waveform mode 2, which is lines
wave_r=0.0
wave_g=1.0
wave_b=0.0
wave_a=1.0
nWaveDots=0
nWaveThick=1
bAdditiveWaves=0

[per_frame_1]
# Per-frame equations
wave_r = 0.0;
wave_g = 1.0;
wave_b = 0.0;
wave_a = 1.0;

[waveform]
# Waveform equations
x = sample;  # Directly map the sample values to the x-coordinate
y = value1;  # Use the waveform's y-coordinate as given
red = 0.0;
green = 1.0;
blue = 0.0;
alpha = 1.0;

Gustavo L Conte

I think i'm on the right track for the issues of my previous flood. I'm sorry for so many messages, but each one had an important milestone maybe and I wanted to report as soon as possible.

These tests now were made with latest visualisations branch, libprojectM version less < than 4, of course. I kinda gave up fixing v4 by now. And Qt6, therefore QOpenGLWidget. I can put this code somewhere but dunno if anybody is interested, its not too big anyway. I believe it can really help debug due to the facts I've mentioned earlier.

Not consuming the buffer
Made again lotsa tests, turns out that the ConsumeBuffer is not functioning at all. With the help of the pulse code that consumes the pulse/pipewire monitor sink, that I mentioned earlier, It could be determined that it works this way, you can see interaction of the preset output and the music, but not with the Clementine code. I'm pretty sure Of this now. I recommend the waveform presets like "Zylot - Crosshair Dimensions (Light of the Ages).milk" to test.
Delay of almost 5 seconds in output
When I could see the music interacting with the output, I've noticed there is a 5 sec delay between the visualization and the beats, etc; this is easily noticed by stopping the music and seeing the waveform stop only 5 seconds later, the contrary obviously happens too.
very Bad performance (at least on my Intel board)
If I comment out the reset function in the drawing method, performance is nice again, otherwise, it consumes 100% of a whole cpu core.
OpenGL issues and deformed output
I've inserted some gl codes and looks like it fixes almost every single plugin output.

I think the one that really fixes is this, on drawBackground

// Set the viewport to match the scene dimensions
    glViewport(0, 0, static_cast<GLsizei>(sceneRect().width() * pixel_ratio_), static_cast<GLsizei>(sceneRect().height() * pixel_ratio_));

also those maybe necessary: (put on the resize window method)

// Adjust the viewport and projection
    glViewport(0, 0, width(), height());

    // Set up the projection matrix
    glMatrixMode(GL_PROJECTION);
    glLoadIdentity();
    glOrtho(0, width(), height(), 0, -1, 1); // Adjust as needed for your coordinate system

    // Switch back to model-view matrix
    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity();

I believe this helps a lot, but I kinda think its not clearing the window as it should. Some plugins make a "trail" in the screen with indicates that.

// Clear the background to avoid flickering
   glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

I put that on the drawing method but maybe its the wrong way IN Qt.

and even still, something is missing, because even if it fixes the majority of plugins, the simplest waveform test I made earlier is still broken.

edit> forgot to mention, I do have an hypothesis to fix the ConsumeBuffer.

The DATA arrives as a char, while add PCM expects an int. I think the reinterpret_cast does not solve this, it only makes us paint a huge wall with a pencil, we need a brush!!

Gustavo L Conte

I've managed to fix the latency of 5 seconds, it was just because of buffer sizes in the pulseaudio capturing code. Now its instant! whee

Soounds like the ResetGL is important too. by putting it back, several presets started to work better including the Simplest waveform milkdrop test. But the "perspective" is different from what the official projectMSDL renders. At least now it isnt just a flashlight...

Gustavo L Conte

WEll, disabling resetGL makes most of the plugins work exactly like projectMSDL
also performance is decent
I saw on the frontend code that they only use resetGL to INIT and when resizing.

Trying hard here to understand the projection matrix, trying both with orthogonal and gluPerspective

But there are those plugins that brake without resetGL, like the simplest waveform test and some others that turn into a white screen. Anyway even those that work only with resetGL on the render loop, they don't render exactly like they should, and most plugins renders real bad.

jonas

@Gustavo-L-Conte
I suspect this might be a Qt bug, or some issue with using projectm and Qt together, although I'm not experienced with OpenGL at all, and I simply lack time to dig more into it since I already have to much work to do with Strawberry. I also tried QOpenGLWindow like the projectm maintainer suggested, but I'm getting similar results.
https://github.com/orgs/projectM-visualizer/discussions/820#discussioncomment-9911992

Gustavo L Conte

i'm trying now to make ConsumeBuffer work as expected, the pulseaudio Capture code helped me a lot to ddebug the presets because some doest not work with none activity.
I've tested forcing opengl profiles via MEsa envvars, it changes the behaviour of some plugins. But there are a few that really wont work and gives just a white screen. I'm comparing to the real projectMSDL and seveeral are functioning ok, it is certainly a bug in way Qt handles the projection view matrix with QOpenGLWidget etc.. or something related to context too.

export MESA_GLSL_VERSION_OVERRIDE=130; 
export MESA_GL_VERSION_OVERRIDE="4.6";

disable resetGL you will see it works much better, several plugins work equal to the original
if we fix ConsumeBuffer, which is not feeding projectM, and remove resetGL, most plugins will work and music will output the visualization properly. I think this bug or whatever it is someday will come to light and then its just a matter of fixing the implementation. There are many other stuff to care about in the application, I agree. This should not go to master, I believe. Its better to wait ppl from llibprojectM -- Maybe if in v4 they find the way with Qt, then it would solve everything, including these version-ballet between v2 v3 v4 etc

Gustavo L Conte

@jonas here's a present for you

if (projectm_) {
  const int samples_per_channel = static_cast<int>(map.size) / sizeof(int) / 4;
  const float *data = reinterpret_cast<float*>(map.data);
  projectm_->pcm()->addPCMfloat_2ch(data, samples_per_channel);
}

Not rdy yet, but on the face of the goal

progressssssssssssssssssssssssssssssssssssssszzzzzzzzz

If i divide by two only half the waveform are rendered, so I tried dividing by 4
Gonna understand that, in the mean time, if you're still going to work on it, change the ConsumeBuffer rooutine ASAP so things work for better debugging....

Gustavo L Conte

this works too, even better, and makes much more sense

const unsigned int samples_per_channel = static_cast<unsigned int>(map.size / sizeof(size_t) / 2);
const float *data = reinterpret_cast<float*>(map.data);
projectm_->pcm()->addPCMfloat_2ch(data, samples_per_channel);

Gustavo L Conte

if (projectm_) {
    short samples_per_channel = static_cast<short>(map.size / sizeof(size_t) / 2);
    const short *data = reinterpret_cast<short*>(map.data);
    projectm_->pcm()->addPCM16Data(data, samples_per_channel);
  }

I think I finally got it. Output is finnally as expected, interacting with the song. We only have to divide by sizeof size_t, in samples_per_channel!!!! Now its working just like when I use the pulseaudio Capture code. (also I've removed the resetGL on the render loop)

Screenshot from 2024-07-05 16-52-13.png
Zylot - Crosshair Dimension (Light of the Ages)

Now we need to fix the projection / OpenGL extensions or whatever is wrong that brakes rendering of plugins.

Gustavo L Conte

Yes! Put this on drawBackground , just after the Init()
also, remove the resetGL

glShadeModel(GL_SMOOTH);
glClearColor(0, 0, 0, 0);
glViewport(0, 0, width(), height());
glMatrixMode(GL_TEXTURE);
glLoadIdentity();
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glDrawBuffer(GL_BACK);
glReadBuffer(GL_BACK);
glEnable(GL_BLEND);

glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glEnable(GL_LINE_SMOOTH);
glEnable(GL_POINT_SMOOTH);
glClearColor(0.0F, 0.0F, 0.0F, 0.0F);
glLineStipple(2, 0xAAAA);

also change the ConsumeBuffer as I sent the post before. Tell me what happes; I'm done here.

Gustavo L Conte

hmm made a few tests and the trick seems to be the
glEnable(GL_BLEND)
but a few plugins that did work, stop working,

anyway, huge progress now!

Gustavo L Conte

I just noticed now that the presets that seem not to work with these new conditions, actually fade to black only when selected in the interface, but when they run as a playlist, they DO WORK AS EXPECTED. I've just selected a few presets that work and doesnt work, and realised they worked when played by projectM itself as a playlist. This is very good news! Means all adjustments in ConsumeBuffer and drawBackground are ALMOST done; its just a matter or understanding why when we select via the menu, some presets do not work. NICE!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Gustavo L Conte

projectm_->initRenderToTexture();
 Resize(sceneRect().width(), sceneRect().height(), container_->devicePixelRatio());

dunno if this change anything, but I forgot to mention that: I've put on the end of the Init() function

PS: its also better to put like 15 seconds to test the playlist. THe presets seem to have kinda of a "initlization" mode and trigger the visualization after about five seconds when fed by the ConsumeBuffer

Gustavo L Conte

Considering what I posted before, about the way selectPreset(index) brakes some presets, making it impossible to preview via the interface, I've made this disgusting, ugly hack to enable preview. The problem is that the indexes are incremented in projectM, so I have to pick index-1 making impossible to get index = 0 which would be the first preset of the list LoLLL

void ProjectMVisualization::SetImmediatePreset(const int index) {

#ifdef HAVE_PROJECTM4
  if (projectm_playlist_instance_) {
    projectm_playlist_set_position(projectm_playlist_instance_, index, true);
  }
#else
  if (projectm_) {
    if ( index <= 0 )
      return;
    Lock(false);
    projectm_->changePresetDuration(1);
    projectm_->selectPresetPosition(index-1);
    // Create a QTimer to call Lock(true) after a delay
        QTimer::singleShot(500, this, [this]() {
            Lock(true);
            projectm_->changePresetDuration(duration_);
        });
  }
#endif  // HAVE_PROJECTM4

}

pretty ugly, but its the proof of concept about what I've posted before. It works. Every single plugin works in the preview now, except the first one (yuck)

PS: the idea here is, instead of setting the preset directly, making projectM make a transition as if it was playing as a playlist queue of the main window. This way, it does not brake the preview, since they all work when projectM manages the transition.

Gustavo L Conte

Thats it! If we put those OpenGL commands in drawBackground,
change the static_cast to englobe both ( map.size / sizeof(size_t) ) /2 in ConsumeBuffer
with these changes in settings:

  s.smoothPresetDuration = 0;
  s.presetDuration = duration_;
  s.shuffleEnabled = false;

and my brand new ugly disgusting hack that now makes the preview work for the first element:

void ProjectMVisualization::SetImmediatePreset(const int index) {

#ifdef HAVE_PROJECTM4
  if (projectm_playlist_instance_) {
    projectm_playlist_set_position(projectm_playlist_instance_, index, true);
  }
#else
  if (projectm_) {
    Lock(false);
    projectm_->changePresetDuration(1);
    if ( index <= 0 )
      projectm_->selectPresetPosition(projectm_->getPlaylistSize());
    else
      projectm_->selectPresetPosition(index-1);
    // Create a QTimer to call Lock(true) after a delay
        QTimer::singleShot(500, this, [this]() {
            Lock(true);
            projectm_->changePresetDuration(duration_);
        });
  }
#endif  // HAVE_PROJECTM4

}

I believe we have a working v3 integration with projectM! I executed the program for more than ten hours without segfaults or bugs.

Screenshot from 2024-07-08 09-08-44.png

jonas

@Gustavo-L-Conte said in Reintegrate projectM Visualizer:

if ( index <= 0 )
  projectm_->selectPresetPosition(projectm_->getPlaylistSize());
else
  projectm_->selectPresetPosition(index-1);

This looks wrong, since the index starts with zero, getPlaylistSize will be too high, needs to do - 1. Also, why are you using index-1 when the index is set? Doesn't IndexOfPreset return the correct index?

Gustavo L Conte

@jonas Its because i need to set the PREVIOUS preset, so that it elapses the transition, thats the only way I found not to bug some presets (that bug only on the preview); After fixing stuff, in the preview interface, some presets bug, i dunno why. But when they are playing normally as a playlist. when projectM is managing the transition, without us forcing with SelectPreset, this bug does not occur.

So I managed to do this ugly hack, that sets the previous preset, unlock, lets projectM do the transition, then "quickly" locks again after the timer

Read my previous posts, it was quite a journey. Thats the only thing missing I believe, to work with v3.

v2 is used on ubuntu and has some issues, i posted previously about that too.

The -1 would segfault, so I force the LAST preset to begin the first, when index = 0 is selected.

Gustavo L Conte

now my elegant BEAUTIFUL hack is gorgeous!

we got ConsumeBuffer consuming
we got drawBackground drawing
we got playlist preview selector selecting and previewing
we aint got no segfault, mon!

YES WE HAVE PROJECTM v3 (and v2 maybe) working

void ProjectMVisualization::SetImmediatePreset(const int index) {

#ifdef HAVE_PROJECTM4
  if (projectm_playlist_instance_) {
    projectm_playlist_set_position(projectm_playlist_instance_, index, true);
  }
#else
  if (projectm_) {
    projectm_->selectPreset(index, true);
    projectm_->changePresetDuration(1);
    projectm_->setPresetLock(false);
    projectm_->selectPrevious(index);
  
    QTimer::singleShot(1500, this, [index,this]() {
      projectm_->setPresetLock(true);
      projectm_->changePresetDuration(duration_);
    }); 
     
  }
#endif  // HAVE_PROJECTM4

}

jonas

@Gustavo-L-Conte
short is the same as int16_t, 2 bytes because the consumed buffer is 16 bit, but size_t is 8 bytes, I'd like to understand why that is correct.
Another thing is that channels are hard-coded, so if the buffer has more then 2 channels, it will be wrong so we should pass channels to ConsumeBuffer

Gustavo L Conte

@jonas I really tried to understand that. I did put size_t because I saw somewhere that map.size type was size_t in gStreamer

The channels I tought it wouldnt be an issue, but thinking about it, it is.

###GstMapInfo

A structure containing the result of a map operation such as Memory.map. It contains the data and size.

struct GstMapInfo {
GstMemory* memory;
GstMapFlags flags;
ubyte* data;
size_t size;
size_t maxsize;
void*[4] userData;
void*[4] GstReserved;
}

its not the buffer, its the SIZE of the buffer, thats why its not 16 bit like the buffer itself!
Maybe thats why